Designing Language Models

When people talk about the compute requirements of a language model, they often seem to be talking about one simple quantity. In practice, a compute figure can describe two very different things: how much arithmetic a job demands, and how fast a machine can do arithmetic. The two are written with similar notation, and that similarity causes a great deal of confusion. This article separates them, then uses each one to make the kind of estimates that planning a model's training and deployment requires.

A FLOP is one floating point operation, and a FLOP count measures how much arithmetic a task demands. Because the count describes a task rather than a machine, it works as a standardized unit for a model's compute requirement. Training a model is a fixed amount of work, so that work is measured in FLOPs. FLOP/s, read as floating point operations per second, is a rate. It describes what a machine can do in one second at its best. A single chip therefore has one FLOP/s rating, while a particular training job needs a separate number expressed in FLOPs. Keeping these two roles apart is the core skill of this section.

Trace the two quantities through a simple question. Suppose someone asks how much compute a model needs. The answer is a total in FLOPs, because the model's training is a fixed job. Now suppose someone asks how fast a GPU is. The answer is a rate in FLOP/s, because the GPU has a capacity per second. A total and a rate answer different questions, and combining them correctly requires a step that connects them, which the next section supplies.

Because FLOP/s is a rate, multiplying it by a length of time converts it into a total count of operations. For large workloads the most useful length is a day. The conversion uses 60 seconds, times 60 minutes, times 24 hours, which gives 86,400 seconds in a day.

A model's parameter count helps estimate the compute resources needed to train and run it, and memory is the most direct link. Each parameter is stored as a number that takes up space. The total storage therefore follows from two values: how many parameters there are, and how many bytes each one occupies. Multiplying the two gives the storage for the parameters alone.

Peak performance is a ceiling that a machine rarely stays at. Utilization measures how much of the maximum compute capacity you can actually use, expressed as a percentage. Utilization is the ratio of delivered performance to peak performance, so it is always between 0% and 100%. Any estimate that uses the peak rating without adjusting for utilization is therefore an optimistic best case.

What counts as good utilization depends on the model, the workload, and the hardware. As a rough guide, the source says that 50% utilization means you are doing okay, and anything above 70% is considered great. A later chapter covers hardware metrics and utilization in more detail.

Now combine the two units. If a training job needs a fixed number of FLOPs, and each machine supplies some FLOP/s, then the time required is the total work divided by the total rate. The total rate is the number of machines multiplied by the effective rate per machine, where the effective rate is the peak rate multiplied by utilization. Utilization below 100% therefore lengthens the training time.

The source summarizes a model's scale with three numbers: parameters, tokens trained on, and FLOPs. Each one is a proxy, meaning an indirect measure of something harder to observe directly. Keeping the three separate prevents you from treating a single figure as a complete description of a model.

These three numbers answer different questions. A model can be described by any one of them, but a fuller picture requires reading them together. Two models with the same parameter count can have very different training exposure and cost, so a single figure rarely tells the whole story.

It is tempting to assume that a larger model is always the better model, but the source describes cases where that assumption fails. In 2022, Anthropic found that more alignment training, discussed in a later chapter on post-training, could lead to models that align less with human preferences, according to the authors of that study.

In 2023, a group of researchers, mostly from New York University, launched the Inverse Scaling Prize to find tasks where larger language models perform worse. Prizes were offered at three levels, and the competition received 99 submissions, of which 11 earned third prizes. The organizers found that larger models are sometimes, though only sometimes, worse on tasks that require memorization and on tasks with strong priors. No second or first prizes were awarded, because the failures appeared on a small test set but none showed up in the real world.