Vocabulary Size and Model Efficiency

Read the beginning of this sentence: 'My favorite color is' followed by a blank. Most people would fill the blank with a color such as blue, and almost never with a word like car. That everyday judgment is what a language model is built to capture. The central question of this article is how a program comes to know which words are likely to appear in a given context.

The word statistical matters here. A language model does not store a dictionary of meanings to be looked up. It stores patterns of likelihood, so that when it is given some context, it can judge which words are probable next. Given the context 'My favorite color is __', a model that encodes English should give 'blue' a higher likelihood than 'car'. The model is therefore best understood as a ranker of probable continuations.

The idea that language has statistical structure is older than computers. In the 1905 story 'The Adventure of the Dancing Men', Sherlock Holmes decodes a sequence of mysterious stick figures using simple statistics of English. Because the most common letter in English is E, he reasons that the most common stick figure must stand for E. The same kind of reasoning, applied with more sophisticated statistics, appears later in the work of Claude Shannon, who used it to decipher enemy messages during the Second World War.

Shannon's 1951 paper 'Prediction and Entropy of Printed English' described how to model English statistically. Many concepts introduced in that paper, including entropy, are still used for language modeling today. Early language models usually covered a single language; today a language model can involve multiple languages.

A language model does not read text as a stream of whole words in the way people do. Its basic unit is called a token. Depending on the model, a token can be a character, a word, or a part of a word, such as the suffix -tion. The process of breaking the original text into tokens is called tokenization.

Read the diagram from left to right. The original phrase enters tokenization and comes out as an ordered sequence of nine tokens. The two changed tokens, the apostrophe and t, are the pieces that came from splitting can't. Every other word in the phrase maps to a single token.

A small set of tokens can build a large number of distinct words, much as a few letters of the alphabet can be combined into many words. The vocabulary is therefore a compact toolkit rather than a list of every word the model could ever meet. The source gives two reference points: the Mixtral 8x7B model has a vocabulary size of 32,000, and GPT-4 has a vocabulary size of 100,256. These numbers belong to specific models and show that vocabulary size differs from one model to another.

Vocabulary size is the count of tokens in this set. It is a property of the model, not of the text being read. A model with a larger vocabulary has more distinct units available to represent text, and a model with a smaller vocabulary must build the same text from a different set of pieces.

The source gives a rough size relationship for GPT-4: an average token is approximately three quarters of a word, so 100 tokens are approximately 75 words. Because the figure is an average, any particular passage can deviate from it.

Prediction starts with context. The preceding words are the input, and the model's stored statistical information about language turns that context into a judgment about which words are likely to come next. The output is not one forced answer. It is a ranking in which some candidates are more probable than others.

Language models come in two main types: masked language models and autoregressive language models. Both are language models in the sense defined earlier, so both encode statistical information about language. The two are distinguished by the information each can use when predicting a token. The difference lies in what each is allowed to look at when it makes a prediction.