Transformer Architecture for Language Foundation Models

Imagine a hospital network that wants an assistant able to read and summarize clinical notes. The notes are written in a regional language and in one specialized medical vocabulary. Before anyone asks which model design to pick, a more basic question comes first: what has the model seen? A model can only become good at patterns that appear in the material it learned from. For foundation models, that material is the training data, and the course treats it as one of the most important factors in how a model performs.

This article follows the chain of decisions that shapes a language foundation model. It starts with training data, then moves to the architecture, the size of the model, the way the model is refined after its first training phase, and the way it produces output. Each choice connects back to the data question, so keep the hospital scenario in mind as you read. By the end you should be able to trace how a batch of token sequences moves through the transformer and how the scale numbers describing that model are related.

Large models need large amounts of training data. Collecting that data is expensive and slow, so providers often use whatever data is available. That practical choice has a consequence: the data may leave gaps for the specific tasks you care about. Curating training data is often necessary for low-resource languages and specialized domains, because the general data a provider can easily gather may contain little of the regional language or medical vocabulary the hospital needs.

Architecture is a deliberate design response to earlier challenges. The course material says the transformer was designed to address certain problems and that it has limitations, and it explores both. The excerpt this article draws on does not list those problems or limitations by name, so this article does not claim a specific list. The practical takeaway is that understanding the transformer means understanding what it handles well and where it falls short.

Architecture is picked before training starts. Two teams that each want a language model may begin differently. One asks which data to gather and how to run the training job. The other first asks what shape the model itself should take. Training usually gets the attention in news coverage, but the architecture and model size decision happens earlier, and it controls how much compute training will require and what the finished model can do.

A model's scale is not captured by one number. It is described with three quantities, and each answers a different question. The number of parameters tells you how many learned values the model has. The number of training tokens tells you how much text the model is trained on. The number of FLOPs needed for training tells you how much computation the training run requires. The three are linked but separate.

Training compute is shaped by two things: the size of the model and the size of the data. A bigger model has more learned values to adjust, and a bigger dataset means more text passes through the training process. Both push total compute upward. The numbers therefore have to be read together rather than one at a time.

The diagram shows the chain of reasoning. Model size and data size both draw on the same compute budget, so increasing one leaves less room for the other. The scaling law is the rule that reads the budget and suggests a balanced allocation between parameters and tokens. The excerpt does not provide a numeric formula for that balance, so the scaling law should be understood as a principle for deciding the split rather than a specific equation.

Models do not process one word at a time in isolation. Input tokens are handled in batches, and the shape of the data tells you how those batches are organized. The course material gives the shape of the input as $N \times T \times 4096$, where $N$ is the batch size and $T$ is the sequence length. The number 4096 is the width of each token's vector. The query, key, and value vectors that the transformer computes from that input have the same shape.

Read the diagram carefully. The input batch enters with three dimensions. Each of the query, key, and value tensors leaves with the same three dimensions. The shape is preserved across these vectors, so the batch size, the sequence length, and the vector width each keep the meaning they had at the input. The excerpt does not describe how the attention computation combines these tensors afterward, so this article stops at the shape of each vector.

A pretrained model is not yet tuned to what people want. Post-training follows pretraining and has two steps: supervised finetuning and preference finetuning. Post-training addresses outputs that do not align with what users want. It narrows the gap between what a pretrained model produces and what people want, but it does not guarantee a perfect match.

A model's output is not a single fixed answer. The model produces outputs through sampling, which makes them probabilistic. Because the output is drawn from possibilities rather than fixed in advance, the same input can lead to different outputs. This brings a benefit, since the model can produce varied wording and options, and a risk, since the output may change from one run to the next in ways a user did not expect.

The source material available for this article names sampling as part of how outputs are produced but does not detail the mechanism further. Treat the benefit and risk described here as the general consequence of probabilistic output, and check the course's fuller treatment of sampling before relying on specifics.