Pre-training Large Models

When people discuss recent progress in AI, one factor comes up again and again: how large the models have become. A large share of the improvement seen in recent years is attributed to increased model size. Because of this, it is hard to talk about foundation models without also talking about how many parameters they have.

This article works through three practical ideas. First, how to read a model's size from its name. Second, how size relates to a model's ability to learn and to its performance within one family of models. Third, how the compute needed to build such models is measured, and why that measurement is incomplete when it is reduced to a count of machines.

A model's parameter count is usually written at the end of its name. To read it, follow the name from left to right; the size is the last piece. The name Llama-13B shows the pattern. The part before the hyphen identifies the model family, here Llama, which is developed by Meta. The part after the hyphen gives the size.

Read the diagram from left to right. The family name tells you which line of models you are looking at. The hyphen marks where the size begins. The number 13 says how many billion parameters that version has, and the letter B confirms that the unit is billions. So Llama-13B tells you directly that this version of Llama has 13 billion parameters.

A parameter count tells you about a model's capacity to learn. Increasing a model's parameters increases that capacity, and the source states that this is likely to produce a better model within the same family. Read the relationship as a chain: the name carries the size, the size sets the capacity, and the capacity is what makes the better outcome possible.

The comparison above keeps the family fixed and changes only the size. That is the scope the source describes. Within one family, a larger parameter count is associated with greater capacity and a likelier better model. The source does not give performance figures for these sizes, so the direction of the effect is the claim to remember, not a specific amount of improvement.

Building a large model is not free. Pre-training large models requires compute, and one way to measure how much is needed is to count the machines involved. The source lists GPUs, CPUs, and TPUs as the kinds of machines counted this way. A team that reports its machine count gives you a first measure of how much compute went into training.

Counting machines is a useful starting measure, but it hides an important detail: machines are not interchangeable. The source gives three examples that are all machines but differ in capacity and cost: an NVIDIA A10 GPU, an NVIDIA H100 GPU, and an Intel Core Ultra Processor. Two entries in a machine count can therefore stand for very different amounts of work.

Read the concept map from the top. Compute is measured in practice by counting machines. That count does not reveal each machine's capacity or cost, and both of those depend on what kind of machine it is. A fair comparison of two efforts therefore needs to account for what each machine is, not only how many there are.