Entropy and Predictability of Data

Imagine you are shown the first part of a sentence taken from a large collection of text and asked to guess the next word. Some sentences make the next word obvious. Others leave you with many reasonable options. A language model faces this same task every time it is trained on a dataset. Its job is to become good at guessing what comes next, and cross entropy is the number that tells you how hard that guessing job is for the model on a given dataset.

Cross entropy is useful in two directions at once. It describes a property of the data itself, namely how predictable the data is. It also describes how well the model's guesses line up with that data. Reading cross entropy well means keeping both ideas in view together, and that is what this article builds toward.

Start with the data alone, before any model enters the picture. If the next item in a dataset is almost always the same, the data is very predictable and there is little uncertainty to deal with. If the next item could be any of many options with roughly equal likelihood, the data is hard to predict. The quantity that captures this predictability is the entropy of the training data, written H(P), where P is the true distribution of the training data.

Notice that entropy never refers to the model. It belongs to the data. Two different models trained on the same dataset face the same entropy value, because nothing about a model changes the true distribution of the data.

Now bring in the model. It has learned its own distribution, called Q, over the possible next items. Each time an item actually arrives, the model has given it some probability. When Q assigns high probability to what really comes next, the model is doing well. When Q spreads its probability onto options that rarely occur, or puts little weight on what really happens, the model pays for it. The figure below traces one next item through this scoring process, from the data to the accumulated cross entropy.

Each outcome therefore contributes one term to the total. An outcome that occurs often in the data and receives little probability from the model adds a large term. An outcome that the model already expects adds a small one. Adding up these terms across the distribution gives the cross entropy, which measures the full cost of predicting the data using the model's beliefs Q rather than the true distribution P.

The key relationship ties the three quantities together. The cross entropy of the model equals the entropy of the data plus a second term that measures how far the model's distribution has drifted from the true one. That second term is the Kullback-Leibler divergence, written $D_{KL}(P \| Q)$.

Suppose the next item is one of two outcomes, A or B. In the true data, A and B each occur half the time, so $P = (0.5, 0.5)$. A model has learned $Q = (0.9, 0.1)$, meaning it strongly believes A is coming. All quantities are measured in bits, which means base-2 logarithms.

Notice what the example reveals. The data has 1 bit of entropy that no model can remove. The model's overconfidence in A adds about 0.737 bits on top. A model that learned Q equal to P would add nothing, and its cross entropy would sit right at the data's entropy.

A tempting shortcut is to assume that the cross entropy between two distributions is the same no matter which one you treat as the true distribution. It is not. Cross entropy is not symmetric. The order of the two arguments tells you whose beliefs are being scored against whose. Each term weights an outcome by the probability from one distribution and takes the logarithm of the probability from the other, so swapping the roles changes both the weights and the logarithms.

The two values differ, so the notation carries real meaning. In language modeling, the order that matters is the one that scores the data P using the model's beliefs Q. That is why the training objective is written with the data's distribution as the weights.