Cross Entropy Loss in Language Modeling

Suppose you are about to train a language model and you have a fixed amount of compute to spend. You now face two linked decisions. The first is how large the model should be. The second is how much text the model should learn from. The two choices are tied together. If you choose a model that is too large, the compute runs out before the model has seen enough data. If you choose one that is too small, the compute is used inefficiently. This article explains how a published scaling rule, the Chinchilla scaling law, connects these two decisions for a given budget, and where that rule stops being a complete answer.

Model performance depends on both the size of the model and the amount of data it trains on. Both push up the amount of compute required, since bigger models and bigger datasets each demand more computation, and computation costs money. The hardware needed is expensive and hard to set up, so compute is often the constraint that actually limits a project. That is why the practical question is usually framed this way: given this much compute, what is the best performance I can get? This framing is the reason the concept is called compute-optimal. A compute-optimal model delivers the best performance achievable for a fixed compute budget, so teams usually start from the budget and work outward.

The loss values used to report how well a language model is doing are measured with cross entropy. The source material this article draws on mentions cross entropy only in passing and states that it is treated in a later chapter, so this article focuses on how budget, model size, and data size relate, and uses cross-entropy values only as a measure of result.

Before looking at the rule itself, follow a single budget through its decisions. Hold the compute fixed and watch what happens to the token count as the parameter count changes. The rule that links the two is a ratio: the number of training tokens D should be roughly 20 times the number of parameters N. Writing that relationship out, we have the following.

The Chinchilla scaling law gives a rule for choosing model size and dataset size once the compute budget is fixed. It came from a 2022 study by DeepMind, which trained 400 language models. Those models ranged from about 70 million parameters to over 16 billion parameters, and they were trained on between 5 and 500 billion tokens. By comparing these runs, the researchers found a consistent relationship: for compute-optimal training, the number of training tokens should be roughly 20 times the number of model parameters. The rule says that model size and token count should be scaled equally. If you double the number of parameters, you should also double the number of training tokens.

The same study also used its experiments to estimate, for each compute budget, the training loss you should expect at the best settings, assuming training is carried out correctly. This is why the rule is useful for planning. It tells you not only how to split a budget between parameters and tokens, but roughly what result that split should produce. The flow below shows how the budget feeds into the two linked decisions.

With the budget held fixed, the two possible failures are easy to describe. Push the parameter count up too far, and the compute is spent on a large model that sees too little text before the budget is exhausted. Push it down too far, and the model is small enough that the compute is used inefficiently. The rule places the balance point where the token count is about 20 times the parameter count. The comparison below sets the three situations side by side.

Read the comparison as a trace. Moving right along the top row, increasing parameters with the budget fixed leaves fewer tokens for the model to learn from. Moving left along the bottom row, shrinking parameters wastes compute. The middle column is the point where the two grow together at the 20-to-1 ratio.

Choosing model size and data is not the only decision a team faces. The performance of a model also depends heavily on its hyperparameters. The source material draws a clear line between the two kinds of value. Parameters are learned during training. Hyperparameters are chosen by users before training. The diagram below contrasts them.

The two kinds of value relate to each other through training. Hyperparameters are set first, the training run then learns the parameters, and the resulting trained model carries those learned parameters forward. The key distinction is who decides the value and when. A parameter is never typed in by a user. A hyperparameter is always a user's choice.

With small models, a common practice is to train several times with different hyperparameter settings and keep the best run. For large models, a single training run is already very expensive, so this trial-and-error approach is rarely feasible. Each tuning trial costs about as much as a full serious run, which means the number of trials a team can afford is very small. Hyperparameters also interact with one another, so the number of combinations to explore grows very quickly.

This creates a practical problem. For many large models, a team may get only one serious attempt at the right hyperparameters. One response is scaling extrapolation, also called hyperparameter transferring. This research area tries to predict good settings for a large model by studying smaller models and extending the pattern upward. A 2022 study by Microsoft and OpenAI reported that hyperparameters could be transferred from a 40 million parameter model to a 6.7 billion parameter model. The approach is still a niche one, because few groups have the resources to study large training runs and because hyperparameters interact so much. Emergent abilities that appear only at scale can also make extrapolation less reliable.

The 20-to-1 rule assumes that acquiring training data is much cheaper than the compute used to train on it. When data itself is expensive to obtain, the optimal split changes, and the same study proposes a separate calculation for that case. The rule also assumes a dense architecture trained mostly on human-generated data. Each of these assumptions is a condition under which the basic answer may need to move. The concept map below shows the factors that pull the choice away from the basic rule.

Inference cost is the adjustment most relevant to a model that will be deployed. The quality reached at training time is only one factor. Some widely adopted models, the Llama family being a prominent example, deliberately use smaller models than the compute-optimal calculation would select for their budget. Their authors could have trained larger models that score better, but smaller models are easier to work with and cheaper to run for inference, and that practical advantage helped them gain broad adoption. To account for this, Sardana and colleagues modified the Chinchilla calculation so that it estimates the best parameter count and pre-training data size while including expected inference demand.

Performance gains also get harder as a model improves, which affects how much a team should expect to spend. Reaching a given level of performance is getting cheaper over time. In one cited example, the compute cost of reaching 93 percent accuracy on the ImageNet dataset roughly halved between 2019 and 2021. Improving a model that is already good, however, remains expensive. Moving accuracy from 90 to 95 percent costs more than moving it from 85 to 90 percent. The same pattern appears in language modeling: a drop in cross-entropy loss from about 3.4 to 2.8 nats requires roughly ten times more training data. Small-looking changes in these loss values can still matter a great deal for downstream applications.