Evaluating Language Model Performance

Imagine you are shown the first part of a sentence taken from a large collection of text and asked to guess the next word. Some sentences make the next word obvious. Others leave you with many reasonable options. A language model faces this same task every time it is trained on a dataset. Its job is to become good at guessing what comes next, and cross entropy is the number that tells you how hard that guessing job is for the model on a given dataset.

Cross entropy is useful in two directions at once. It describes a property of the data itself, namely how predictable the data is. It also describes how well the model's guesses line up with that data. Learning to read cross entropy means keeping both ideas in view together. This article builds that skill step by step, tracing the numbers at each stage so you can see exactly where each part of the score comes from.

Start with the data alone, before any model enters the picture. If the next item in a dataset is almost always the same, the data is very predictable and there is little uncertainty to deal with. If the next item could be any of many options with roughly equal likelihood, the data is hard to predict. The quantity that captures this predictability is the entropy of the training data, written $H(P)$, where $P$ stands for the true distribution of the training data.

Read the formula one outcome at a time. For each possible next item x, take its true probability P(x), multiply it by the base-2 logarithm of that probability, add all of these terms together, and flip the sign so the result is positive. Outcomes that are very likely contribute little uncertainty, while spread-out probabilities produce a larger total.

Now bring in the model. It has learned its own distribution, which we call Q, over the possible next items. When Q assigns high probability to what actually comes next, the model is doing well. When Q spreads its probability onto options that rarely occur in the data, or puts little weight on what really happens, the model pays for it. Cross entropy measures this full cost: the difficulty of predicting the data, judged using the model's own beliefs Q rather than the true distribution P.

The steps show why the weights come from the data while the logarithms come from the model. Outcomes that happen often in the data are weighted heavily, and the model is penalized most when it gives those frequent outcomes little probability. This is the mechanism behind a confident wrong guess raising the score.

The key relationship ties the three quantities together. The cross entropy of the model equals the entropy of the data plus a second term that measures how far the model's distribution has drifted from the true one. That second term is the Kullback-Leibler divergence, written $D_{KL}(P \| Q)$, which measures how far the model's distribution Q is from the true distribution P.

Read the equation as a sum of two costs. The first term, H(P), is the uncertainty the data carries, and no model can remove it. The second term, the KL divergence, is the extra cost caused by the model's distribution differing from the truth. Whatever the model does, it cannot push cross entropy below the entropy of the data. The KL term is the only part the model can shrink through learning.

This gives a helpful way to read cross entropy. You can think of a model's cross entropy as its approximation of the entropy of its training data. A cross entropy that sits well above the data's entropy signals a gap that the model has not yet closed.

Suppose the next item in a dataset is one of only two outcomes, call them A and B. In the true data, A and B each occur half the time, so P = (0.5, 0.5). A model has learned Q = (0.9, 0.1), which means it strongly believes A is coming. All quantities are measured in bits, which means base-2 logarithms are used throughout.

Notice what the example reveals. A model that learned Q equal to P would add nothing to the 1 bit of data entropy, and its cross entropy would sit right at the data's entropy. Here the model's miscalibration is visible as the extra 0.737 bits.

A tempting shortcut is to assume that the cross entropy between two distributions is the same no matter which one you treat as the true distribution. It is not. Cross entropy is not symmetric. The cross entropy written H(P,Q) differs from the cross entropy written H(Q,P). The order of the arguments tells you whose beliefs are being scored against whose.

The order tells you whose beliefs are being scored against whose. H(P,Q) asks how well the model Q explains data distributed as P. H(Q,P) swaps those roles. Because the two expressions weight the terms differently, they produce different numbers, so writing the arguments in the wrong order changes the meaning of the score.