Sampling and the Probabilistic Nature of Models

Imagine two teams that each want to build a language model. The first team starts by asking which data to gather and how to run the training job. The second team starts earlier, by asking what shape the model itself should take. Training gets most of the attention in news coverage, but a major decision happens before training begins: choosing the architecture and the size of the model. That decision controls how much compute training will require and what the finished model can do.

This article focuses on the architecture side of that decision. Before you can judge a model, you need a working picture of two things: the transformer, which is the design that now dominates language-based foundation models, and the scale numbers used to describe how big a model is. As you read, keep one question in mind. When a model is large and capable, which part of that result comes from its architecture, and which part comes from how much it was scaled?

The distinction matters. A dominant design wins because it solves a set of problems well enough that most builders reach for it first. That popularity does not mean every problem is solved, and it does not mean the design is free of tradeoffs. A builder who treats a dominant default as the only option may miss the places where it falls short.

The course material says the transformer was designed to address certain problems and that it has limitations, and it explores both. The excerpt this article draws on does not list those problems or limitations by name, so this article does not claim a specific list. What the material does make clear is that the transformer is a deliberate design response to earlier challenges. Understanding it means understanding what it handles well and where it falls short.

A model's scale is not captured by one number. It is described with three quantities: the number of parameters, the number of training tokens, and the number of FLOPs needed for training. Each one answers a different question about the model. Parameters tell you about the model's size. Training tokens tell you how much data the model is trained on. FLOPs tell you how much computation training requires.

The three numbers are linked but separate. A model can have many parameters and still be trained on a small amount of data. Two models with the same parameter count can need very different amounts of compute depending on how much data they see. This is why reading only the parameter count tells you about the model's size but not about how expensive it was to build.

Training compute is shaped by two things: the size of the model and the size of the data. A bigger model has more learned values to adjust during training. A bigger dataset means more text passes through the training process. Both push the total amount of compute upward. Compute is therefore not a fixed property of the model alone. It grows with the model and also with the data the model is trained on.

This is where the scaling law comes in. Given a fixed compute budget, a scaling law helps determine the best combination of number of parameters and number of training tokens. The question it answers is practical. If you can spend only so much compute, should you build a larger model trained on fewer tokens, or a smaller model trained on more tokens? Scaling laws give builders a principled way to make that split instead of guessing.

The source excerpt does not give a numeric formula for the recommended balance. The diagram shows only the relationships: size and data both feed the compute requirement, and the scaling law takes a compute budget and returns a recommended split. Do not assume a specific formula beyond that relationship unless your source provides one.

Models do not process one word at a time in isolation. Input tokens are handled in batches, and the shape of the data tells you how those batches are organized. The course material gives the shape of the input as a three-dimensional arrangement written $N \times T \times 4096$. Here $N$ is the batch size, meaning how many sequences are processed together, and $T$ is the sequence length, meaning how long each sequence is. The 4096 is the width of each token's vector.

The query, key, and value vectors that the transformer computes from that input have the same shape. So the batch keeps its three dimensions as it moves through the Q, K, and V step. The diagram below traces this path. The source excerpt does not detail the steps that come after these vectors are formed, so the diagram stops there.