Foundation Models and Language Models
Training cross entropy is tied to how well the model predicts its own training data. When the model assigns high probability to what actually comes next, cross entropy goes down. The metric is therefore a readout of predictive quality, not an arbitrary score.
A Demo That Ships Too Early
Imagine a team that has just built an application on top of a foundation model. The model answers questions reasonably well in the first demo, so the team ships it. Weeks later they change a prompt, swap in a newer model, or add a new kind of user request. They have no reliable way to say whether the application got better or worse.
This situation is common, and it points to a wider problem. Evaluation, the work of measuring how well an AI system performs, receives less investment than many other parts of AI engineering. This article looks at what that gap looks like, why it matters, and how a family of language modeling metrics gives you a foothold for reasoning about model quality.
Where the Evaluation Gap Shows Up
Interest in evaluation has grown, but it still trails interest in the rest of the AI engineering pipeline. The gap is visible in research attention, in tooling, and in everyday practice. Each of these is a separate kind of evidence, and each points in the same direction.
Following One Prediction Through Training
To see why language modeling metrics are useful, start with the core idea behind a language model. It encodes statistical information about a language: how likely a given token is to appear in a given context. Consider the context 'I like drinking' followed by a blank. A model that has absorbed enough statistics about English should rank the next word as more likely to be tea than charcoal.
The more statistical information a model captures, the better it becomes at predicting the next token. In machine learning terms, a language model learns the distribution of its training data. The better it learns that distribution, the better it predicts what comes next in the training data, and the lower its training cross entropy.
Training cross entropy is tied to how well the model predicts its own training data. When the model assigns high probability to what actually comes next, cross entropy goes down. The metric is therefore a readout of predictive quality, not an arbitrary score.
Four Metrics, One Measurement
Most autoregressive language models are trained with cross entropy, or with its close relative, perplexity. The metrics used to guide language model development have changed little since Claude Shannon's 1951 paper on the prediction and entropy of printed English, which popularized the field. When you read papers and model reports, you will also meet bits-per-character (BPC) and bits-per-byte (BPB). Both are variations of cross entropy.
The four metrics are closely related. If you know the value of one, you can compute the other three, provided you have the necessary information. You do not need to treat them as four unrelated numbers. They are four views of the same underlying measurement of how well a model predicts a sequence. The diagram below shows how each view is reached from the shared measurement.
In this relation, H is the cross entropy and PPL is perplexity. The formula is the standard link between the two, and it is why perplexity is described as exponentiated cross entropy. The exponent is measured in the same natural-log units as the cross entropy, so the base used when the loss was computed matters when you convert.
What do you think happens?
If a model's cross entropy drops from 3.0 to 2.5, does its perplexity go up or down?
Reveal answer
Down
Perplexity is e raised to the cross entropy, and e raised to a larger number gives a larger result. A smaller cross entropy therefore gives a smaller perplexity. Lower cross entropy and lower perplexity always move together.
Same Model, Different Units
The four metrics differ mainly in what they are normalized by. Cross entropy and perplexity are computed per token, while BPC and BPB are normalized per character and per byte. Because the same total loss is divided by a different count in each case, the same model can produce numbers that look very different on the page. Comparing a per-token number with a per-byte number directly is therefore misleading unless you convert between them.
| Metric | Normalized by | What it reads as |
|---|---|---|
| Cross entropy | Per predicted token | Average loss for each token the model predicts |
| Perplexity | Per predicted token | Exponentiated cross entropy, interpreted as an effective number of choices |
| Bits per character | Per character | Average information needed to encode each character |
| Bits per byte | Per byte | Average information needed to encode each byte |
The four metrics share one measurement but use different denominators.
Tracing a Perplexity Value
Turning a cross entropy into a count of choices
Suppose a model's cross entropy, measured in nats, is about 3.0. What perplexity does that correspond to, and how many equally likely next-token choices is the model effectively hedging between? Then work backward from a perplexity of 20.
- 1Apply the relation
Perplexity equals e raised to the cross entropy, so PPL = e to the power of 3.0.
- 2Evaluate
e cubed is about 20.1, so the perplexity is roughly 20.
- 3Interpret
A perplexity of about 20 means the model behaves as if it were choosing uniformly among roughly 20 equally likely next tokens.
- 4Work backward
For a perplexity of exactly 20, the cross entropy is the natural logarithm of 20, which is about 3.0. The two directions agree.
- 5Check the direction
If cross entropy falls to about 2.5, perplexity falls to about 12.2, so the model hedges among fewer choices.
A cross entropy near 3.0 corresponds to a perplexity near 20, and a drop in cross entropy shrinks the effective number of choices.
Predicting Downstream Behavior
Foundation models grew out of language models, and many of them still have a language model as a main component. For these systems, the performance of that language model component tends to track closely with how the foundation model performs on the applications built on top of it. This connection is why language modeling metrics are worth understanding even if the system you build is far from a text generator. A rough grasp of these metrics gives you a way to anticipate downstream performance.
These metrics are also not limited to natural language. They can be used for any model that generates sequences of tokens, including tokens that are not text. This matters when your model's outputs are sequences of other kinds of symbols, because the same reasoning about cross entropy and its variants still applies.
Mistakes When Comparing Metrics
Common mistake Comparing a per-token perplexity with a per-byte BPB as if they were the same unit
What it looks likeModel A reports a perplexity, Model B reports a BPB, and the team picks whichever number is smaller.
Why it failsEach number is normalized by a different count, so they are not on the same scale.
Do this insteadConvert both to a common normalization before comparing, or compare only metrics computed the same way.
Common mistake Treating a training metric as a measure of production quality
What it looks likeLoss on the training corpus looks excellent, so the team assumes users will get the same results.
Why it failsA metric describes predictions on the data it was computed on, and production data may differ from training data.
Do this insteadCheck how similar the evaluated data is to the data you will serve.
Common mistake Treating perplexity as a percentage of correct answers rather than as an effective count of choices
Consequences of Eyeballed Evaluation
Ad hoc evaluation based on eyeballed outputs and personal prompt choices can work at a project's start, when the goal is only to get something running. It is not sufficient for repeated iteration, because changes cannot be compared reliably when the test set is informal and the judgments are impressionistic. The flow below traces how that failure typically unfolds.
Keep a test set that reflects what the application actually needs, rather than a list of favorite prompts. Use the same test set each time you change the prompt, the model, or the request types, so that before and after results can be compared directly.
Checking Your Reasoning
A teammate reports that Model X has a cross entropy of 2.0 per token and Model Y has a perplexity of 9. Without converting to a common unit, can you decide which model predicts its data better? Explain how you would decide, using the relation between cross entropy and perplexity.
Hints
- Perplexity is e raised to the cross entropy, so convert Model X's cross entropy into a perplexity.
- Both values are per token here, so the comparison is valid once they are on the same metric.
Key Takeaways
- AI evaluation receives less investment than other parts of AI engineering, visible in research attention, tooling counts, and everyday practice.
- Many foundation models contain a language model component, so language modeling metrics give a rough but useful signal about downstream performance.
- Cross entropy, perplexity, BPC, and BPB are four views of one measurement of how well a model predicts a sequence; perplexity is e raised to the cross entropy.
- The metrics differ mainly in their normalization, so compare them only after converting to a common unit, and remember that training metrics may not describe your production data.
- Ad hoc, eyeballed evaluation can work at the start of a project but cannot reliably show whether later changes made an application better or worse.