Masked and Autoregressive Language Models

Read the sentence 'My favorite color is ___' and try to fill the blank. Most people would answer with a color such as blue, and almost no one would answer with a word like car. That everyday judgment is what a language model is built to capture. A language model is a program that learns how likely words are to appear in a given context, so that it can judge which words are probable in a particular position.

This article answers one core question: how does a program come to know which words are likely to appear in a given context? Along the way it introduces the basic unit a model works with, called a token, and the two main families of language models, masked and autoregressive.

The word statistical matters. A language model does not store a dictionary of meanings to be looked up. It stores patterns of likelihood. When it is given some context, it can judge which words are probable next. For the context 'My favorite color is __', a model that encodes English should give 'blue' a higher likelihood than 'car'. The model's skill is this ranking of probable continuations, not any single fixed answer.

The idea that language has statistical structure is older than computers. In the 1905 story 'The Adventure of the Dancing Men', Sherlock Holmes decodes a sequence of mysterious stick figures using simple statistics of English. Because the most common letter in English is E, he reasons that the most common stick figure must stand for E. The same kind of reasoning, applied with more sophisticated statistics, appears later in the work of Claude Shannon, who used it to decipher enemy messages during the Second World War.

Shannon's 1951 paper 'Prediction and Entropy of Printed English' described how to model English statistically. Many concepts from that paper, including entropy, are still used for language modeling today. Early language models usually covered a single language. Today a language model can involve multiple languages.

A language model does not read text as a stream of whole words the way people do. Its basic unit is called a token. Depending on the model, a token can be a character, a word, or a part of a word, such as the suffix -tion. The process of breaking the original text into tokens is called tokenization.

The source gives a concrete case. GPT-4, the model behind ChatGPT, breaks the phrase 'I can't wait to build AI applications' into nine tokens. Notice how the contraction is handled: can't does not survive as one piece. It is split into can and a second piece made of the apostrophe and t. Tokens are therefore not the same thing as the words a person would count on a page.

A natural question is why a model should use tokens at all, when it could use whole words or single characters. The source answers with a balance. Characters give a very small set of units, but they carry little meaning on their own. Whole words carry meaning, but there are very many of them, and a made-up word can be unknown to the model entirely. Tokens sit between the two: there are fewer units than words, yet each unit still carries more meaning than a single character.

A small set of tokens can build a large number of distinct words, much as a few letters of the alphabet can be combined into many words. The vocabulary is therefore a compact toolkit, not a list of every word the model could ever meet. The vocabulary size is the number of entries in that toolkit, so it sets how many distinct tokens the model can represent. These two figures belong to specific models and show that vocabulary size is not the same across models.

Token counts and word counts are also related through an average. The source states that for GPT-4 an average token is approximately three quarters of a word, so 100 tokens are approximately 75 words. Working backwards, a passage of 400 tokens would be roughly 300 words. This is an average, so any particular passage can deviate from it.

Before naming the two model types, follow one prediction through a model. Start with a partial sentence and ask what the model is allowed to see when it fills the next position. Think about this for a moment before reading the answer.

Language models come in two main types: masked language models and autoregressive language models. Both are language models in the sense defined earlier, so both encode statistical information about language. The two are distinguished by the information each can use when predicting a token. The difference lies in what they are allowed to look at when they make a prediction.

An autoregressive model generates text by repeating one step. It predicts a next word, appends that word to the sequence, and then feeds the longer sequence back in to predict the word after it. The loop below follows a short generated example that starts from the words 'The cat sat'.

Each pass uses a slightly longer sequence than the last, because every predicted token is added to the context before the next prediction. This is why the word autoregressive fits: each new output is regressed on what the model has already produced.