The GPT Model Architecture

Before any text can be generated, the model has to exist and be placed in a known state. This chapter begins with a listing that builds that starting point. Its key feature is that it does not define the model's internals. It passes one dictionary to the GPTModel class and receives an object back. The same class can produce a small or a large model depending only on the numbers stored in that dictionary.

The dictionary is called GPT_CONFIG_124M, and it is the single place where the model's shape is recorded. Each key names one setting, and each value fixes that setting for the model that gets built. As you read the dictionary, ask two questions: which lines decide how large the model is, and how much text it can look at in one pass. You will return to those lines when you trace the generation process.

The dictionary holds seven keys. Only one of them, context_length, is changed from the previous chapter. The other six keep the settings used in chapter 4, so the model is the same architecture with a shorter window of text. Among the settings this article names directly, n_layers controls how many transformer blocks are stacked, and context_length controls how many tokens the model can look at in one pass.

The context length defines how much text the model sees in a single pass. The chapter moves it from 1,024 tokens to 256 tokens while keeping the rest of the architecture the same. The source's stated reason is practical: shortening the context reduces the computational demands of training. That reduction is what makes training feasible on a standard laptop computer.

Text generation with a GPT model is a cycle that starts with text and ends with text. It moves through three stages. In the first stage, the tokenizer converts input text into a series of token IDs. In the second stage, the model receives those token IDs and produces logits. A logit vector represents the probability distribution over every token in the vocabulary. In the third stage, those logits are converted back into token IDs, and the tokenizer decodes those IDs into human-readable text.

The stages are easy to confuse because two of them produce numbers. The model's output is not text, and it is not a token ID either. It is a vector of scores over the vocabulary. Only after that vector is turned back into a token ID can the tokenizer produce a word or a piece of a word.

The chapter adopts the generate_text_simple function from chapter 4 and adds two helper functions, text_to_token_ids and token_ids_to_text. Together they form the boundary between the human side and the numeric side of the cycle. The helper names describe their direction. text_to_token_ids moves from text toward the numeric side. token_ids_to_text moves from numbers back toward text. Keeping that direction in mind makes it easier to decide which helper belongs at a given point in the cycle.

When a generated result looks wrong, the most useful question is which stage produced the unexpected value. Check the stages in order: the token IDs after encoding, the logits from the model, and then the text after decoding. Each stage boundary is a place where a problem can hide, so checking in order tells you where to start debugging.

Earlier, a stand-in version of a GPT model showed how data enters and leaves the model, with its inner parts left as placeholders. This section replaces those placeholders with the real building blocks, the TransformerBlock class and the LayerNorm class. Once they are in place, the model can be assembled into a fully working version of the 124-million-parameter GPT-2 model. Read the architecture from input to output and it becomes a pipeline. Tokenized text is first turned into token embeddings. Positional embeddings are then added, so the model knows where each token sits in the sequence. The combined tensor enters the stack of transformer blocks. After the last block, a final layer normalization step adjusts the result, and a linear output layer produces the scores used to predict the next token.

The number of repetitions is not written into the block itself. It is controlled by a configuration entry called n_layers, which lives in the GPT_CONFIG_124M dictionary. For the 124-million-parameter GPT-2 model, that entry is set so the transformer block is repeated 12 times. The largest GPT-2 model uses 48. Changing the configuration value changes the depth of the model without changing the code of the block. Only the middle section of the pipeline is repeated, which is why the number of blocks is the main lever for changing the model's depth.

The final answer the model produces for each position is a set of values that spans every token the model knows. To see how that set is formed, follow the data from the last transformer block. Its output is first adjusted by the final layer normalization step, which prepares the representation. The linear output layer then maps that representation into a space with one dimension per vocabulary entry, 50,257 in the 124-million-parameter case. Each dimension is a score for one token. The next token is chosen from that set of scores.