Cross Entropy in Language Models

Suppose two language models both report a cross entropy of 6 bits. It is tempting to conclude that they are equally good at the same job. That conclusion rests on an assumption that often fails: that each model's token means the same amount of text. A model that treats each word as a token and a model that treats each character as a token can both report 6 bits per token, yet those 6 bits cover very different amounts of text.

This article follows the path a score takes as it is rewritten into a fairer form. You will trace what one token covers, convert bits per token into bits per character, convert again into bits per byte, and finally read that number as a statement about compression.

Cross entropy is measured in bits per token. The bit count tells you how much the model spends to predict each token, but it says nothing about how many characters that token stands for. Two tokenizers can split the same sentence into very different numbers of pieces, so the same per-token cost hides different amounts of text.

Here the two models carry identical scores of 6 bits per token, but Model A's token covers about 6 characters while Model B's covers about 1. Model B's 6 bits are spread across far less text, so each character costs it far more. Reading the two numbers side by side as if they measured the same thing would be a category error.

Bits-per-character (BPC) rescales the per-token figure into a per-character figure. Take the number of bits the model spends on each token and divide it by the average number of characters that a token contains. Because every model can be described in terms of characters, BPC gives a common basis for comparing models that split text in different ways.

BPC still has a weakness, and it comes from character encoding. The number of bits used to store a character depends on the encoding scheme. Under ASCII, each character is encoded using 7 bits. Under UTF-8, a character can take anywhere from 8 to 32 bits. Two models could report identical BPC values while the underlying text occupies different amounts of storage, which makes BPC an unreliable yardstick when the encodings differ.

Bits-per-byte (BPB) fixes this by anchoring the measure to the byte, the unit of the original training data rather than the character. BPB is the number of bits a language model needs to represent one byte of that data. Because the reference unit is the stored byte, the measure accounts for the encoding scheme, which is why the source calls BPB a more standardized metric than BPC.

The reason these metrics matter is their link to cross entropy. The source states that cross entropy tells us how efficient a language model will be at compressing text. Cross entropy is measured in bits, and BPB expresses that same cost against the bytes of the original text. A lower BPB therefore means fewer bits are spent on each original byte, so the model can represent the text more compactly.

The metrics in this article are easy to apply mechanically, and that is where most errors come from. The three mistakes below are the ones that most often lead to wrong conclusions.