Training Objectives for Language Models

Imagine two prediction tasks built on the same kind of text. In the first, a model sees the sentence 'My favorite __ is blue' with one word hidden and must supply the missing word. In the second, a different model sees only the start of a sentence, 'My favorite color is', and must say what word should come next. Both tasks ask a model to guess a token from the language around it, yet the information each model is allowed to use is not the same. That difference in what the model can see is the central idea behind the two main types of language models.

The rest of this article traces what each model can see at each step of its training objective. Once you can say which tokens are visible and which are hidden, the differences in applications follow naturally.

A masked language model is trained to predict missing tokens anywhere in a sequence, using context from both before and after each gap, as in filling in the blank. The training objective is not about continuing a sentence. The model is handed a sequence with some tokens hidden and must recover what was hidden.

In the diagram, the masked position is the only one the model must fill. Every other token is available as evidence, and the tokens on both sides of the gap feed into the same prediction. Reading the diagram from the gap outward, the left and right neighbors are both inputs to one decision.

A well-known example of the masked family is BERT, which stands for bidirectional encoder representations from transformers. Masked language models were commonly used for nongenerative tasks such as sentiment analysis and text classification. They are also useful when a task depends on the overall context of a passage.

An autoregressive language model is trained to predict the next token using only the preceding tokens. Here the model is restricted to the left side of the position it is predicting. Given 'My favorite color is __', it must produce the word that continues the sentence, and it may not look ahead to see what the sentence eventually says.

Read the steps from top to bottom. At each step the visible context is everything written so far, and the position being predicted is the one after it. Nothing to the right of the unknown position is available, so the window of visible context only ever extends forward as the sequence grows.

Once a prediction is made, it can be appended to the sequence, and the model can then predict the token after that. This is the mechanism behind the ability to continually generate one token after another. Each new token becomes part of the context for the next prediction.

The source describes this feedback loop as the reason autoregressive models suit generation: the model's job mirrors the way a sentence is written from left to right. The source does not describe how the next token is chosen from the model's predictions, so this article does not go into that step.

The two model types differ in two linked ways: which tokens they may condition on, and what their training objective asks them to produce. A masked model fills a gap using context on both sides. An autoregressive model extends a sequence using only what came before.

The source states that autoregressive models are the models of choice for text generation, and for this reason they are far more popular than masked language models. Masked models remain the natural fit when the task is to understand or label a passage using its full context.

A language model works from a fixed, finite vocabulary, yet it can combine those vocabulary items into an unlimited number of possible outputs. A model that can produce open-ended outputs in this way is called generative, and this is the origin of the term generative AI. Language models are therefore a central example of generative AI, because their outputs are not limited to a fixed list of answers.