Cross Entropy

When a language model is evaluated, one of the most frequently reported figures is perplexity. Reading that figure with confidence requires knowing what it measures and how it connects to cross entropy, the quantity it is most often confused with. This article starts from that confusion, tests it, and then builds the correct relationship from the ground up.

Imagine you are shown the first part of a sentence from a large collection of text and asked to guess the next word. Some sentences make the next word obvious, while others leave you with many reasonable options. A language model faces this same task every time it is trained on a dataset. Its job is to become good at guessing what comes next, and cross entropy is the number that tells you how hard that guessing job is for the model on a given dataset.

The idea is useful in two directions. It describes a property of the data itself, namely how predictable the data is. It also describes how well the model's guesses line up with that data. Learning to read cross entropy means keeping both of these ideas in view at once.

Start with the data alone, before any model enters the picture. If the next item in a dataset is almost always the same, the data is very predictable and there is little uncertainty to deal with. If the next item could be any of many options with roughly equal likelihood, the data is hard to predict. The quantity that captures this predictability is the entropy of the training data, written $H(P)$, where $P$ is the true distribution of the training data.

Notice that entropy never refers to the model. It belongs to the data. Two different models trained on the same dataset would face the same entropy value, because nothing about the model changes the true distribution of the data.

Now bring in the model. It has learned its own distribution, $Q$, over the possible next items. When $Q$ assigns high probability to what actually comes next, the model is doing well. When $Q$ spreads its probability onto options that rarely occur, or puts little weight on what really happens, the model pays for it. Cross entropy measures this full cost: the difficulty of predicting the data, judged using the model's own beliefs $Q$ rather than the true distribution $P$.

Read the equation as a sum of two costs. The first term, $H(P)$, is the unavoidable amount of uncertainty the data carries. The second term, $D_{KL}(P \| Q)$, is the Kullback-Leibler divergence, the extra cost caused by the model's distribution differing from the truth. Whatever the model does, it cannot push cross entropy below the entropy of the data. The KL term is the only part the model can shrink through learning.

Suppose the next item in a dataset is one of only two outcomes, call them A and B. In the true data, A and B each occur half the time, so $P = (0.5, 0.5)$. A model has learned $Q = (0.9, 0.1)$, meaning it strongly believes A is coming. Measure everything in bits, which means base-2 logarithms.

Notice what the example reveals. The data has an entropy of 1 bit that no model can remove. The model's overconfidence in A adds roughly 0.737 bits on top. A model that learned Q equal to P would add nothing, and its cross entropy would sit right at the data's entropy.

Perplexity keeps the same measurement as cross entropy but expresses it as an exponential. Given a dataset with true distribution $P$, the perplexity is the base 2 exponential of the entropy $H(P)$. For a language model with learned distribution $Q$ on that same dataset, the perplexity is the base 2 exponential of the cross entropy $H(P,Q)$.

Because the exponential grows quickly, a cross entropy of 2 bits corresponds to a perplexity of 4, a cross entropy of 3 bits corresponds to a perplexity of 8, and each additional bit doubles the perplexity. The relationship is one of exponentiation, not of relabeling.

The reason exponentiation is the right operation becomes clear when you think about options. Uncertainty about the next token means there are several candidates the model could pick. A useful test case is a model trained to encode four position tokens perfectly, and then asked to predict a position inside a square. Each bit distinguishes two unique values, so the number of equally likely options grows as a power of 2 when cross entropy grows in bits.

Higher perplexity indicates more uncertainty, meaning more possible options for the next token. A perplexity of 4 means the model faces the equivalent of four equally likely choices, as in the diagram above. A model that is uncertain across many more options will report a larger perplexity.

So far the base has been 2, which matches the idea that each bit distinguishes two unique values. The unit, however, is a convention rather than a law of nature. Popular machine learning frameworks, including TensorFlow and PyTorch, measure entropy and cross entropy in nats. A nat uses the base of e, the base of the natural logarithm. When nats are the unit, perplexity becomes the base e exponential of cross entropy.

A tempting shortcut is to assume that the cross entropy between two distributions is the same no matter which one you treat as the true distribution. It is not. Cross entropy is not symmetric. The cross entropy of Q with respect to P, written H(P,Q), differs from the cross entropy of P with respect to Q, written H(Q,P). The order of the arguments tells you whose beliefs are being scored against whose.

You can verify this with the same two-outcome numbers. Scoring the data P using the model's beliefs Q gave about 1.737 bits. Swapping the roles gives H(Q,P) = -(0.9 log2 0.5 + 0.1 log2 0.5) = 1 bit. The two values are different, so the notation carries real meaning.

The standard convention places the true data distribution first, so $H(P,Q)$ means the data is weighted by its own probabilities while the model's beliefs supply the scoring. Keeping the order fixed is what makes the split into data entropy plus KL divergence valid.