Tokenization and Its Effect on Model Evaluation

Suppose two language models both report a cross entropy of 6 bits. It is tempting to conclude that they are equally good at the same job. That conclusion depends on an assumption that often fails: that each model's token means the same amount of text. A model that treats each word as a token and a model that treats each character as a token can both report 6 bits per token, yet those 6 bits cover very different amounts of text.

This article follows one idea through the whole discussion. Bits per token is a per-token figure, so it only means something when you know how much text a token holds. The two metrics introduced to fix this problem are bits-per-character (BPC) and bits-per-byte (BPB). Together they connect cross entropy to a practical question: how much can a model compress text?

A tokenization method cuts a stretch of text into spans. Each span becomes one token, and the spans can be long or short depending on the method. The same stretch of text can therefore be cut into a small number of long tokens or a large number of short tokens. When a model reports a fixed number of bits for each token, the bits are being spread over whatever span that token happens to cover.

To see this concretely, take a 12-character text, abcdefghijkl. Tokenizer A cuts it into six tokens of 2 characters each: ab, cd, ef, gh, ij, and kl. Tokenizer B cuts the same text into three tokens of 4 characters each: abcd, efgh, and ijkl. Both tokenizers describe exactly the same text. Only the spans differ.

Notice what changed. Tokenizer A uses six tokens and Tokenizer B uses three, yet the text is identical. Each token in A covers half as many characters as each token in B. If every token costs 6 bits, then A spends more total bits on the same text, and the bits it spends are spread over a smaller span per token.

Now apply the same measurement to both tokenizers. Assume each token costs 6 bits, as in the opening example. A raw bits-per-token comparison says the two models are identical. Converting to bits per character shows the difference.

Bits-per-character rescales the per-token figure into a per-character figure. The procedure is simple. Take the number of bits the model spends on each token and divide it by the average number of characters that a token contains. Because every model can be described in terms of characters, BPC gives a common basis for comparing models that split text in different ways.

In the source's own example, a model spends 6 bits per token and each token averages 2 characters, so the BPC is $\frac{6}{2} = 3$ bits per character. The pipeline below shows the same steps as a flow: two inputs feed one division, and the division produces the BPC.

BPC removes the dependence on token size, but it still has a weakness, and the source points to its cause: the number of bits used to store a character depends on the character encoding scheme. Under ASCII, each character is encoded using 7 bits. Under UTF-8, a character can take anywhere from 8 to 32 bits.

Bits-per-byte fixes this by anchoring the measure to the byte, the unit of the original training data rather than the character. BPB is the number of bits a language model needs to represent one byte of that data. Because the reference unit is the stored byte, the measure accounts for the encoding scheme, which is why the source calls BPB a more standardized metric than BPC.

To move from BPC to BPB, the bits-per-character value is scaled by how much of a byte each character occupies. The source's example uses ASCII, where each character is 7 bits. Seven bits is $\frac{7}{8}$ of a byte, because a byte has 8 bits.

The reason these metrics matter is their link to cross entropy. The source states that cross entropy tells us how efficient a language model will be at compressing text. Cross entropy is measured in bits, and BPB expresses that same cost against the bytes of the original text. A lower BPB therefore means fewer bits are spent on each original byte, so the model can represent the text more compactly.

The baseline is the original data itself. Each original byte has 8 bits. A model with a BPB of 3.43 represents that byte using only 3.43 bits. Because 3.43 is less than half of 8, the model can compress the original training text to less than half its original size.