Predicting the Next Token in Language Models

Imagine someone describing a location to you one word at a time, while you try to anticipate each word before it arrives. Sometimes the next word is almost obvious. At other times you have many plausible options, and any of them could show up. This article is about putting a number on that difference. Language-modeling work uses a quantity called entropy for exactly this purpose.

The way to understand entropy is to notice what changes when a language offers more or fewer possible tokens to choose from. The rest of this article follows that idea step by step: first by tracing a small language, then by reading the number that describes it, and finally by working through a count of tokens.

Consider a small invented language whose only job is to name positions inside a square. Each token in the language tells you where something sits. The design choice that matters most is how many tokens the language contains, because that choice controls how specific each token can be.

A language that names only two broad regions gives coarse information: each token says whether a position is in the upper or lower part of the square. A language that names four finer regions gives more specific information, because each token pins down a more precise position. The four-token language is the one that carries more information per token. It also asks more of the system that stores and transmits those tokens.

The source states the difference directly: the tokens in the richer language carry more information, and they also need more bits to represent them. Keep this pairing in mind, because the rest of the article depends on it.

The word average is doing real work. A language is made of several possible tokens, and each one may carry a different amount of information. Entropy condenses that collection into one number that describes the typical token. It is a summary of the whole language, not a property of any single token.

Entropy links two ideas that might otherwise seem separate. The first is how much a token tells you. The second is how many bits a token costs to represent. The source gives two consequences: a higher entropy means each token carries more information, and a higher entropy means more bits are needed to represent a token.

Entropy can also be read as a measure of difficulty. The source describes it as a measure of how difficult it is to predict what comes next in a language. Under this reading, the two-token language and the four-token language differ in how hard it is to guess the next token. Choosing among two possibilities is easier than choosing among four.

A lower-entropy language is therefore more predictable, and a higher-entropy language is harder to predict. The key picture is the shape of the probability spread across possible next tokens. When one token dominates, the spread is peaked and the next token is easy to anticipate. When many tokens are about equally likely, the spread is flat and the next token is hard to anticipate.

The square example gives a concrete pattern: two tokens give entropy 1, and four tokens give entropy 2. The source does not spell out a general rule, but for tokens that are equally likely, the entropy in bits equals the base-2 logarithm of the token count. This rule reproduces both numbers from the example, so it is a useful way to read the pattern.