Comparative Evaluation and Preference Models

As AI models grow stronger, the damage a single failure can cause grows with them, so evaluating these models matters more over time. Evaluating open-ended, powerful models is hard, and many teams fall back on human judgment. Humans in the loop are useful for sanity checks, and in many cases human evaluation is essential. Automatic evaluation is where this material focuses, and language modeling metrics are one of its foundations, because many foundation models contain a language model component.

Two of the language modeling metrics, perplexity and cross entropy, have a reputation for being confusing. Many people find them hard to interpret. This article takes a different route from a typical explanation: it starts with the units of text that the metrics count, then builds up to entropy, cross entropy, and finally perplexity. The order matters, because a number produced by a metric only means something once you know what was being counted.

These are three different ways to cut up the same text, and the choice is not cosmetic. The source notes that tokenization affects language modeling performance. It points out that you might want to decide whether cats and cat should count as two separate tokens, or whether will not and won't should count as two separate tokens. Each decision changes what the model is asked to predict and what the metric measures.

Read the diagram column by column. In the character column, the word is four tokens long. In the word column, it is a single token. In the part-of-word column, it becomes two tokens: a stem and a suffix. The text is identical in all three cases, yet the count of tokens differs, and any metric averaged over tokens will reflect that difference. This is the practical reason tokenization has to be understood before comparing numbers from two different systems.

Suppose two evaluation teams both report a per-token score for the same passage. Team A tokenized the passage into characters. Team B tokenized it into whole words. Team B's passage has far fewer tokens, because each word is one unit where Team A has several. Nothing about the passage changed, only the unit being counted.

Cross entropy builds on entropy, so start there. When Claude Shannon introduced entropy in 1951, the tokens he worked with were characters. Entropy, in his framing, is a statistical parameter that measures how much information is produced on average for each letter of a text in a language. If the language is translated into binary digits in the most efficient way, entropy is the average number of binary digits needed per letter of the original language.

The unit in Shannon's definition is a letter, which is a character-level token. That is why it is a natural starting point for language modeling metrics that count tokens. Entropy is about the average information per unit, not the total length of the text. A long text and a short text can therefore have the same entropy if they carry information at the same average rate.

A useful way to hold entropy is as a rate: information per unit of text. It answers how much new information each letter carries on average, not how much information the whole text contains. Two passages of very different lengths can share the same rate.

The source describes cross entropy as a measure of information production per letter, and it lists the entropy introduced by Claude Shannon in 1951 as its prerequisite. Cross entropy is therefore the same kind of average-per-unit quantity, used in the context of language modeling. There, a model predicts the next token, and those predictions are judged against the text that actually appears.

The source pack does not spell out the arithmetic that links cross entropy to the model's probabilities, so this article does not derive it. What it does establish is the framing: cross entropy is an average per unit of text, and the unit depends on how the text was tokenized. A cross entropy value from one tokenizer is therefore not automatically the same kind of number as a cross entropy value from another.

Entropy is often expressed in binary digits, which uses base 2 logarithms. The source notes that many people prefer the natural logarithm instead, because it has properties that make the math easier. One concrete property it cites is the derivative of the natural log, which keeps the calculus simple. This is why the same information measure can appear with different logarithm bases in different sources.

Perplexity is a language modeling metric, and the source lists it alongside cross entropy as one of the metrics many people find confusing. The source says the chapter includes a section on interpreting these metrics so they can be used in evaluation and data processing. Hold that frame: perplexity is a tool for judging a language model, and it should be read in the context of how the text was tokenized and what the model was asked to predict.

Perplexity measures how well a model predicts the next token. It does not directly measure whether the model will succeed at a task built on top of it. The source makes this distinction explicit: there is a strong correlation between language modeling performance and downstream performance, but language modeling performance does not fully explain downstream performance.