Cross-entropy loss

A language model is evaluated by how well its predicted probability distribution matches the words that actually appear in a dataset. At each position, the model spreads probability across candidate next tokens, and the evaluation asks how much weight it gave to the token that really came next. Cross-entropy loss is one measure used for this evaluation. Perplexity is another way to read the same result.

Perplexity expresses the model's uncertainty about which token comes next. The clearest way to see that uncertainty is in the shape of the probability distribution. When the model puts most of its probability on one candidate, the distribution is peaked and the model is confident. When it spreads probability across many candidates, the distribution is flat and the model is uncertain. The diagram below compares these two cases for the same context.

In the confident column, almost all the probability sits on one token, so a model that is right would have a small loss and a perplexity near 1. In the uncertain column, the probability is shared among four tokens, so the model is less sure which one comes next, and its perplexity is larger. The shape of the distribution, not any single number on its own, is what perplexity summarizes.

Perplexity is calculated from loss. The source states that it is obtained by taking the exponential of the loss, written in PyTorch as torch.exp(loss). In mathematical notation, if the loss is $L$, then the perplexity is $e^{L}$. Because the exponential undoes the logarithm inside the loss, perplexity returns the result to a scale that is easier to read as a degree of uncertainty.

Read the pipeline from left to right. Each correct next token receives a probability. A higher probability means a smaller negative log-probability, so a confident correct prediction contributes little to the loss. The contributions are averaged across the sequence to give the cross-entropy loss. Exponentiating that loss gives the perplexity. A lower perplexity means the model's predictions sit closer to the distribution of the words actually found in the dataset.

The following example shows the calculation in code. Suppose a model gives the correct next token a probability of 0.05 at every position, which is what a model would do if it were choosing uniformly among 20 tokens. The per-token loss is then the negative natural logarithm of 0.05, which equals the natural logarithm of 20. Taking the exponential of that loss returns 20.

The same code works for any sequence of correct-token probabilities. Only the list of probabilities changes. The average of the negative log-probabilities is the loss, and the exponential of that loss is the perplexity.

Perplexity can be read as an effective vocabulary size. A perplexity of 20 means the model is about as uncertain about the next token as it would be if it were choosing among 20 equally likely tokens. This is a helpful way to think about uncertainty, but it is only an analogy. It does not mean that the model is literally considering exactly 20 candidate tokens, nor does it mean the model has any particular set of 20 tokens in mind.

Two distributions can have the same perplexity while looking different in detail. One model might put most of its probability on a few tokens with a long tail of small values, while another spreads the probability more evenly. Perplexity compresses both into one number that reflects overall uncertainty, which is why it should be read as an effective size rather than as a list of candidates.