Foundation Models and Parameters
A model's parameter count is usually written at the end of its name, as in Llama-13B.
Why Model Size Keeps Coming Up
When people discuss recent progress in AI, one factor comes up again and again: how large the models have become. A large share of the improvement seen in recent years is attributed to increased model size. Because of this, it is hard to talk about foundation models without also talking about how many parameters they have.
This article follows one model name through three questions. First, how do you read a model's size from its name? Second, how does size relate to a model's ability to learn? Third, how is the compute needed to build such a model measured, and why is that measurement incomplete? Keep the name Llama-13B in mind as you read, because it is the running example.
What do you think happens?
A team says it used ten machines to pre-train a model. Can you tell from that number alone how much compute was spent?
Reveal answer
No, the count alone does not settle it
Machines differ in capacity and cost. Ten machines of one kind may have far more capacity than ten machines of another kind, so the number of machines is only part of the picture.
Reading the Size From a Model Name
A model's parameter count is usually written at the end of its name. To read it, follow the name from left to right and look at the last piece. In the name Llama-13B, the part before the hyphen identifies the model family, here Llama, which is developed by Meta. The part after the hyphen gives the size.
Tracing the size out of Llama-13B
What does the name Llama-13B tell you about the model?
- 1Start with the full name
The name is Llama-13B. The size is written at the end, so begin at the right-hand side.
- 2Split at the hyphen
The hyphen divides the name into two parts: Llama on the left and 13B on the right.
- 3Identify the family
The left part, Llama, names the model family. The source notes this family is developed by Meta.
- 4Read the number
The right part, 13B, gives the size. The number 13 is the count.
- 5Decode the unit
The letter B stands for billion. So 13B means 13 billion parameters.
Llama-13B is the Llama family's version with 13 billion parameters. The name states the parameter count directly.
What a Parameter Count Says About Capacity
A parameter count tells you about a model's capacity to learn. Increasing a model's parameters increases that capacity. The source states that this is likely to produce a better model within the same family. Read this as a chain: the name carries the size, the size sets the capacity, and the capacity is what makes the better outcome possible.
The comparison above is a pattern, not a measurement. The source does not give benchmark numbers, so do not assume a particular size gap produces a particular gain. It says only that more parameters increase capacity and are likely to produce a better model within the same family. The phrase within the same family is important: the comparison is fair when the models share a family, because the family determines much of the design.
When comparing models, first check that they belong to the same family. Only then does the parameter count offer a meaningful comparison of capacity and likely performance.
Measuring Pre-Training Compute by Counting Machines
Building a large model is not free. Pre-training large models requires compute, and one way to measure how much is needed is to count the machines involved. The source lists GPUs, CPUs, and TPUs as the kinds of machines counted this way. Counting is a useful starting measure.
The count hides an important detail: machines are not interchangeable. The source gives three examples of machines: an NVIDIA A10 GPU, an NVIDIA H100 GPU, and an Intel Core Ultra Processor. Each is a machine, yet each has a different capacity and a different cost. A count that treats them as identical loses that information.
Why ten machines is not a complete answer
A team reports that it used ten machines for pre-training. Why can that number not tell you the compute spent?
- 1Take the count
The reported figure is ten machines. That is one measure of compute.
- 2Ask what kind of machine
Suppose the ten machines are one kind. The count does not say which kind, so the capacity is unknown.
- 3Compare two cases
Ten machines of one kind may have far more capacity than ten machines of another kind. The same count can describe very different amounts of compute.
- 4Account for each machine
A fair comparison needs to know what each machine is, not only how many there are.
Ten is a starting measure. The total compute depends on the capacity of the machines counted, so the count must be read together with the kind of machine.
Mistakes When Reading Model Sizes
Common mistake Reading the number in a model name without its unit
What it looks likeSeeing Llama-13B and treating 13 as thirteen parameters.
Why it failsThe letter B stands for billion, and the name gives the parameter count directly.
Do this insteadRead the number together with its unit: 13B means 13 billion parameters.
Common mistake Assuming a bigger parameter count always wins, regardless of family
What it looks likeComparing models from two different families only by their size.
Why it failsThe source ties the size-to-performance relationship to models within the same family.
Do this insteadCompare parameter counts within one family, and treat cross-family comparisons with more caution.
Common mistake Treating a machine count as a full measure of compute
What it looks likeConcluding that two teams used equal compute because each reported ten machines.
Why it failsMachines differ in capacity and cost, so the same count can describe different amounts of compute.
Do this insteadAsk what kind of machines were counted, and read the count alongside their capacity.
Common mistake Claiming a specific performance gain from a specific size increase
What it looks likeStating that a larger model is exactly a set percentage better.
Why it failsThe source says larger capacity is likely to produce a better model, and gives no benchmark figures.
Do this insteadDescribe the relationship as likely, not as a measured quantity.
Practice Reading Names and Compute
Read the name Llama-13B again and state, in one sentence, the model family and the parameter count. Then explain why a reported count of ten machines would not, on its own, tell you the compute used.
Hints
- The family is the part before the hyphen, and the size is the part after it.
- The letter B stands for billion.
- Think about whether all machines have the same capacity and cost.
Summary
At a glance
- A model's parameter count is usually written at the end of its name, so Llama-13B means the Llama family's version with 13 billion parameters.
- Increasing a model's parameters increases its capacity to learn, which is likely to produce a better model within the same family.
- Pre-training large models requires compute, which can be measured by counting machines such as GPUs, CPUs, and TPUs.
- Machines differ in capacity and cost, so a machine count alone does not fully describe the compute used.