Interpreting Perplexity Values

Suppose two people report perplexity numbers for what appears to be the same language model, and the numbers differ widely. Before you decide which result is better, you need to know what the number was measured on and how it was computed. Perplexity is a single value, but it is shaped by several factors: the structure of the data, the size of the vocabulary the model predicts over, and how much previous text the model is allowed to look at.

Perplexity measures a model's predictive accuracy. A lower value means the model predicts the text more accurately, and a higher value means the model is more uncertain about what comes next. Because of this, a perplexity number is not a fixed property of a model alone. It depends on the dataset being predicted and on the exact conditions under which the measurement was made.

Every factor in this article works through one mechanism: uncertainty. When a model has more uncertainty about which token comes next, perplexity rises. When the surrounding text makes the next token easy to anticipate, perplexity falls. Each factor is simply a different way the model's uncertainty can grow or shrink. Keep this single idea in mind as the three factors are introduced.

The first factor is the structure of the data. More structured data is more predictable, and predictability is exactly what perplexity measures. Structure means that the text follows rules that tell the reader what to expect. Those rules give the model clues about upcoming tokens, so its uncertainty drops.

The source's main example is HTML code. If a model sees an opening head tag, it can predict that a matching closing tag should appear nearby. Everyday text does not offer that kind of reliable signal. Because the model is more confident about HTML, its expected perplexity on HTML code should be lower than its expected perplexity on everyday text.

The second factor is vocabulary size, meaning how many possible tokens the model must choose among at each step. When there are more possible tokens, it is harder to identify the correct one, so the model is more uncertain and perplexity goes up. The source illustrates this with two texts. A model's perplexity on a children's book will likely be lower than the same model's perplexity on War and Peace, because the children's book uses a narrower range of words.

The same logic applies to the unit being predicted. For the same dataset, character-based perplexity, which measures how well the model predicts the next character, will be lower than word-based perplexity, which measures how well it predicts the next word. The reason is that the number of possible characters is smaller than the number of possible words. Choosing among a small set of characters is an easier task than choosing among a very large set of words.

The third factor is context length, the amount of previous text the model can condition on when predicting the next token. The more context a model has, the less uncertainty it has about the next token, so its perplexity tends to be lower. The principle is old. In 1951, Claude Shannon evaluated his model's cross entropy by using it to predict the next token conditioned on up to 10 previous tokens.

At the time the source was written, a model's perplexity could typically be computed and conditioned on between 500 and 10,000 previous tokens, and possibly more. The upper limit is set by the model's maximum context length. Context length is not unlimited: it grows as models are built to handle more text, but it always stops at the model's own maximum.

Beyond the three factors, the source explains why perplexity matters in practice. Perplexity is a good proxy for a model's capabilities. If a model is bad at predicting the next token, its performance on downstream tasks will also likely be bad. For that reason, perplexity is useful across an AI engineering workflow, not only for guiding the training of language models.

The source supports this with OpenAI's GPT-2 report, which shows that larger models, which are also more powerful models, consistently give lower perplexity on a range of datasets. In the LAMBADA column, the perplexity falls from 35.13 at 117M parameters to 8.63 at 1542M parameters. The link is a rough signal, not a guarantee: it depends on comparing models on a shared test set under the same conditions.