Hyperparameters and Training Configuration

Imagine you are about to train a language model and you have a fixed amount of compute to spend. Compute is the computation you pay for when a model learns from text, and for large models it is often the constraint that actually limits a project. The hardware needed is expensive and hard to set up, so teams usually frame the practical question as: given this much compute, what is the best performance I can get?

This article follows that outward path. It starts with the two linked decisions every team faces on a fixed budget, then walks through the Chinchilla scaling rule that connects them, then separates the values a model learns from the settings a person chooses, and finally covers the situations in which the rule must be adjusted.

On a fixed budget you face two decisions at once. The first is how large the model should be, meaning how many parameters it has. The second is how much text the model should learn from, meaning how many training tokens it sees. The two are linked because model performance depends on both, and both push up the amount of compute required.

The two failure modes pull in opposite directions. If the model is too large, the compute runs out before it has seen enough data. If the model is too small, the compute is used inefficiently. The diagram below compares these two allocations under the same budget.

The Chinchilla scaling law gives a rule for choosing model size and dataset size once the compute budget is fixed. It came from a 2022 study by DeepMind, which trained 400 language models. Those models ranged from about 70 million parameters to over 16 billion parameters, and they were trained on between 5 and 500 billion tokens. By comparing these runs, the researchers found a consistent relationship: for compute-optimal training, the number of training tokens should be roughly 20 times the number of model parameters.

The same study also estimated, for each compute budget, the training loss you should expect at the best settings, assuming training is carried out correctly. This makes the rule useful for planning. It tells you not only how to split a budget, but roughly what result that split should produce.

A training configuration contains two different kinds of values, and mixing them up causes confusion. Parameters are the values the model learns during training. Hyperparameters are the settings that are chosen by users before training, and they govern how that learning happens. The diagram below shows how the two sit within a training configuration.

The distinction matters for the budget discussion. The Chinchilla rule decides how size and data should be balanced, but the hyperparameters still have to be chosen for the run. The performance of a model depends heavily on them, so a correct size and data split alone does not guarantee a good result.

With small models, it is common to train several times with different hyperparameter settings and keep the best run. The loop below shows that cycle: choose settings, train, evaluate, then adjust and repeat.

For large models the same cycle becomes rarely feasible. A single training run is already very expensive, so a team may get only one serious attempt at the right hyperparameters. The hyperparameters also interact with one another, so the number of combinations grows very quickly, and a team cannot simply test each one in isolation.

One research direction tries to work around this. Scaling extrapolation, also called hyperparameter transferring, studies smaller models and extends the pattern upward to predict good settings for a large model. One 2022 study by Microsoft and OpenAI reported that hyperparameters could be transferred from a 40 million parameter model to a 6.7 billion parameter model. The approach is still niche, because few groups have the resources to study large training runs. Emergent abilities that appear only at scale can also make extrapolation less reliable.

The Chinchilla rule is a starting point with stated assumptions. It was developed for dense models trained mostly on human-generated data, and it assumes that acquiring training data is much cheaper than the compute used to train on it. It does not automatically carry over to sparse architectures such as mixture-of-experts models, nor to training on synthetic data. Adapting the calculation to those settings is an active research area.

Two conditions push the decision away from the pure training-compute answer. The first is inference demand. For a model that will be deployed, the quality reached at training time is only one factor. Some widely adopted models, the Llama family being a prominent example, deliberately use smaller models than the compute-optimal calculation would select for their budget. Their authors could have trained larger models that score better, but smaller models are easier to work with and cheaper to run for inference, and that practical advantage helped them gain broad adoption.

To account for this, Sardana and colleagues modified the Chinchilla calculation so that it estimates the best parameter count and pre-training data size while including expected inference demand. The lesson is that the optimal size depends on what you are optimizing for: training quality alone, or training plus the ongoing cost of serving the model.

The second condition is expensive data. When data itself is costly to obtain, the optimal split changes, and the same study proposes a separate calculation for that case. The original ratio should not be applied unchanged when data is the scarce resource.