Scaling Laws for Foundation Models

The transformer is the dominant architecture for language-based foundation models, and it is chosen before training begins. Dominant means the common default, not the only design.

Programming

The Decision Before Training

Training often gets the attention, but an important decision comes earlier: what architecture and model size to use. The transformer is the dominant architecture for language-based foundation models. Dominant means it is the common default, not the only design people have tried. Choosing the architecture and size helps set the design of the model and affects how much compute training will require.

Worked example

Two teams planning a language model

Compare a team that begins by asking what data to gather and how to run training with a team that first asks what shape the model should take.

1
Notice the order
The first team starts with data and the training job. The second considers the model's design before training begins.
2
Identify the earlier decision
Choosing the architecture and model size is a major decision made before training. It affects the compute training will require and what the finished model can do.

Result

Planning a training job is not the whole design process: the architecture and size must also be considered before training starts.

Which major design decision is made before the training job begins?

Three Measures of Scale

A model's scale is not captured by a single number. It is described using the number of parameters, the number of training tokens, and the number of FLOPs needed for training. These quantities are linked, but they measure different things: model size, the amount of training data, and training compute.

What does each scale measure, and what question does it help answer?

The distinction matters when comparing models. A parameter count tells you about model size, but it does not tell you how much training data the model saw or how expensive training was. Models with the same number of parameters can require different amounts of compute if they are trained on different amounts of data.

How a Compute Budget Shapes the Choice

Training compute depends on both model size and data size. A larger model has more learned values to adjust, while a larger dataset means more text passes through training. Both contribute to the compute requirement. That is why parameters and training tokens should be considered together rather than treating either number as a complete description of scale.

How do model size and training data relate to compute, and what does a scaling law help decide under a fixed budget?

A scaling law is used to help choose the balance between parameters and training tokens for a given compute budget. The practical choice is whether to spend the budget on a larger model trained on fewer tokens or a smaller model trained on more. The source gives this decision-making purpose, but not a numeric formula for calculating the balance, so the diagram represents a relationship rather than a numerical recipe.

Best practice

When evaluating a claim about a model's scale, ask which quantity is being discussed. Parameters describe model size; training tokens describe data size; training FLOPs describe compute needed. Do not use one as a substitute for the other two.

Following a Batch Through Q, K, and V

A transformer handles input tokens in batches. The input shape is $N \times T \times 4096$. Here, $N$ is the number of sequences processed together, $T$ is the sequence length, and $4096$ is the width of each token's vector. Query, key, and value vectors are computed from the input and share that same shape.

What happens to the batch dimensions when the input is represented as query, key, and value vectors?

Read the three dimensions separately: $N$ counts sequences in the batch, $T$ counts positions in each sequence, and $4096$ is the vector width. In this source-supported trace, the dimensions do not change as the input becomes query, key, and value vectors. The material here does not specify the shapes of attention scores or later outputs, so those shapes cannot be inferred from this description alone.

Common Misreadings

Common mistake
Treating the parameter count as the whole measure of scale.
What it looks like
A model has a large number of parameters, so its training cost must be known.
Why it fails
Parameter count describes model size, but training compute also depends on the amount of data processed.
Do this instead
Consider parameters, training tokens, and training FLOPs as linked but separate quantities.
Common mistake
Assuming a dominant architecture is the only possible design.
What it looks like
The transformer is dominant, so no other architecture has been tried.
Why it fails
Dominant means the common default, not the only design.
Do this instead
Describe the transformer as the dominant architecture for language-based foundation models without claiming it is the sole design.
Common mistake
Assuming architecture alone determines model behavior.
What it looks like
Two models with the same architecture must behave the same.
Why it fails
Behavior also depends on what a model was trained on and how it was adjusted afterward.
Do this instead
Consider architecture alongside training data and later adjustments.

Training data is a practical constraint as well as a scale measure. Acquiring large amounts of data can be expensive and time-consuming. Providers may use available data, which can leave gaps for particular tasks. So a model's architecture and scale do not, by themselves, guarantee that its training data covers the work you care about.

Practice the Scale Decision

MEDIUM

A team has a fixed training compute budget and is comparing a larger model trained on fewer tokens with a smaller model trained on more tokens. What role does a scaling law play in this decision, and which three quantities should the team keep distinct?

Hints

Separate model size from data size and training compute.
The scaling law helps determine a balance for the fixed budget; it does not mean the three quantities are identical.

Your attempt (not saved)

A scaling law helps the team decide how to balance parameters and training tokens under its compute budget. Parameters measure model size, training tokens measure data size, and FLOPs measure compute needed for training. Without a numeric scaling relationship, the exact best combination cannot be calculated from these facts alone.

Key Takeaways

The transformer is the dominant architecture for language-based foundation models, and architecture and model size are considered before training begins.
Parameters, training tokens, and training FLOPs describe model size, data size, and training compute; they are linked but not interchangeable.
Both model size and data size contribute to training compute. A scaling law helps choose their balance for a compute budget.
The input batch and its query, key, and value vectors share shape $N \times T \times 4096$, where $N$ is batch size and $T$ is sequence length.

More concepts starting with S