Model Size and Compute Requirements

Suppose two language models come from the same family, and one is noticeably stronger on every task you try. What made the difference? A tempting answer is that the stronger one is simply bigger. That answer is partly right, but it hides several other forces that shape how well a model turns its design into useful behavior.

This article walks through the main factors: how many parameters a model has, how those parameters are used, how much data it was trained on, what that data looked like, and how recently the model was built. The goal is to see each factor clearly enough to reason about a real comparison. The practical side of the topic is reading a model's size from its name, relating size to a model's ability to learn, and understanding how the compute needed to build such models is measured.

A model's parameter count is usually written at the end of its name. Parameters are the adjustable numbers that training shapes, and the parameter count is often the first figure people look at when they compare models. Follow the name from left to right and the size is the last piece.

Take the name Llama-13B. The part before the hyphen identifies the model family, here Llama, which is developed by Meta. The part after the hyphen gives the size: 13B means the version of Llama with 13 billion parameters. The letter B stands for billion, so the name states the parameter count directly.

A parameter count tells you about a model's capacity to learn. Increasing a model's parameters increases that capacity, and within one model family a larger model generally performs better. The chain runs like this: the name carries the size, the size sets the capacity, and the capacity is what makes a better outcome possible.

Size is not the only way to advance. As builders of models learn better training methods, each new generation of a family tends to beat older generations of the same size. The clearest illustration in the source compares two Llama models on the MMLU benchmark. Llama 3-8B, released in 2024, outperforms Llama 2-70B, released in 2023, even though the older model has far more parameters. A small model from a newer generation can therefore beat a much larger model from an older one, so generation and training approach must be considered alongside size.

Capacity also needs material to fill it. A large model trained on too little data can underperform a much smaller model that saw more. The source gives a deliberately extreme case: imagine a 13-billion-parameter model whose entire training set is the single sentence 'I like pineapples.' Such a model has enormous capacity and almost nothing to learn from, so it will perform far worse than a much smaller model trained on a substantial body of text.

The parameter count also tells you roughly how much hardware is needed. Each parameter is stored as a number that takes a fixed number of bytes. If you know that number and the count, you can estimate the minimum memory required to hold the model while it runs inference, which means producing outputs rather than learning.

In this expression, P is the number of parameters and b is the number of bytes used to store each parameter. The result is a lower bound on the memory needed to hold the weights. Storage at 32 bits uses 4 bytes per parameter, 16 bits uses 2 bytes, and 8 bits uses 1 byte. The same parameter count therefore has different footprints depending on how each value is stored.

A raw parameter count can be misleading when the model is sparse. A sparse model is one in which a large share of the parameters hold the value zero. Those zeros do not contribute to the computation in the same way nonzero values do, so counting them as full parameters overstates the working size. The source states the consequence directly: a sparse model can require less compute than a dense one, even when the sparse model is larger on paper.

One popular sparse design is the mixture-of-experts model, usually abbreviated MoE. An MoE model divides its parameters into separate groups, and each group is called an expert. The model does not run every expert on every token. Only a subset of the experts is active for processing any given token, which is why the model can hold a large total parameter count while spending much less compute on each token.

The diagram is illustrative: it shows eight experts and two active for a single token. The source does not fix how many experts are active at once, only that a subset is engaged. The key idea is the split between the two totals. The total parameters describe everything stored in the model, while the compute for one token depends on the experts actually used.

Data quantity is the second major lever. Early on, researchers measured a dataset's size by counting training samples. That stops working for language models, because a training sample might be a sentence, a Wikipedia page, a chat conversation, or a whole book, and a book carries far more material than a sentence. The better yardstick is the number of tokens, the unit a language model actually operates on. Counting tokens tells you roughly how much the model has the opportunity to learn from its data.

Token counts depend on the tokenizer, because different models split text differently and the same dataset can produce different token totals. Even so, token counts give a far more useful comparison than sample counts. Meta's Llama models show the trend: Llama 1 was trained on 1 trillion tokens, Llama 2 on 2 trillion, and Llama 3 on 15 trillion. Modern language models are trained on datasets measured in trillions of tokens.

Two numbers are easy to confuse here: the number of tokens in the dataset, and the number of tokens the model sees during training. Training tokens count every pass over the data, so a dataset of a given size can produce a larger training token count when the same material is passed through more than once. The dataset total describes what you have; the training total describes how much learning happened.

Training token counts do not scale neatly with parameter counts. Reading across several models in the source shows that a bigger parameter count does not automatically come with a bigger token budget. Chinchilla, with far fewer parameters than several of the others, was trained on a much larger number of tokens. Size and token budget must be compared separately.

Quantity is not the only thing that matters. The source names three goals for training data, which it calls the three golden goals: quantity, quality, and diversity. A large dataset can still be a poor teacher if its contents are weak.

The source illustrates this with RedPajama-v2, an open dataset from Together that contains 30 trillion tokens. That is an enormous volume, but because it is made of indiscriminate content, the portion that counts as high quality is much smaller. A dataset with many tokens and low quality may teach a model less than a smaller, cleaner, more varied one. The source points to a later chapter for a fuller treatment of quality and diversity.

Building a large model is not free. Pre-training large models requires compute, and one way to measure how much is needed is to count the machines involved. The source lists GPUs, CPUs, and TPUs as the kinds of machines counted this way.

Counting is a useful starting measure, but it hides an important detail: machines are not interchangeable. An NVIDIA A10 GPU, an NVIDIA H100 GPU, and an Intel Core Ultra Processor are all machines, yet each has a different capacity and a different cost. Suppose a team reports that it used ten machines for pre-training. That number cannot tell you how much compute was really spent, because ten machines of one kind may have far more capacity than ten machines of another kind.

A fair comparison therefore needs to account for what each machine is, not only how many there are. The source's section on compute ends by naming the scaling law, described as the concept behind building compute-optimal models. The source gives this idea only a heading and a one-line association, so this article does not go further than that. What you can take from it is the direction: model size, machine counts, and compute are considered together, with the aim of building models efficiently for the compute they use.