Compute-Optimal Model Design

When people discuss recent progress in AI, one factor comes up again and again: how large the models have become. A large share of the improvement seen in recent years is attributed to increased model size. Because of this, it is hard to talk about foundation models without also talking about how many parameters they have.

This article works through the practical side of that idea. You will learn how to read a model's size from its name, how size relates to a model's ability to learn, and how the compute needed to build such models is measured. Each of these steps builds on the one before it, so keep the chain in mind as you read: name, then size, then capacity, then the compute needed to build the model.

A model's parameter count is usually written at the end of its name. Follow the name from left to right and the size is the last piece. Take the name Llama-13B. Before you read the explanation below, try to answer the question yourself: which part of this name tells you the size of the model?

Read the name as a sentence with three parts. The part before the hyphen names the family. The part after the hyphen gives the size, and it sits at the end of the name. The letter B is a unit, so the name tells you the parameter count directly: 13B means the version of Llama with 13 billion parameters.

A parameter count tells you about a model's capacity to learn. Increasing a model's parameters increases that capacity, and the source states that this is likely to produce a better model within the same family. The comparison is only meaningful inside one family, because the name tells you the family and the size together.

Read the relationship as a chain. The name carries the size, the size sets the capacity, and the capacity is what makes the better outcome possible. The chain is a direction of reasoning, not a formula: more parameters give more capacity to learn, and that capacity is what the source links to better results within a family.

Building a large model is not free. Pre-training large models requires compute, and one way to measure how much is needed is to count the machines involved. The source lists GPUs, CPUs, and TPUs as the kinds of machines counted this way. A machine count is a useful starting measure, because it tells you roughly how much hardware a team committed to the work.

Counting is a useful starting measure, but it hides an important detail: machines are not interchangeable. An NVIDIA A10 GPU, an NVIDIA H100 GPU, and an Intel Core Ultra Processor are all machines, yet each has a different capacity and a different cost. Two counts that look the same can therefore describe very different amounts of hardware.