Building a GPT Model from a Configuration Dictionary

Loading a pretrained GPT-2 model is not one action. It is a sequence of steps that depend on each other. First the settings have to be decided, because a model cannot be built without them. Then the pretrained weights have to be downloaded, because they cannot be loaded before they exist locally. Finally the weights have to be loaded into a model whose shape matches them. This article follows one short script that performs these steps in order. You will predict what its key variables hold, then trace where a result could go wrong.

The goal is not to memorize the script line by line. The goal is to look at a run, point to the step where the outcome stopped matching what you expected, and explain why.

The script runs in a fixed order. It first imports three names: a download helper, the GPTModel class, and a weight-loading function. It then defines a base configuration dictionary, defines a second dictionary of model sizes, chooses a model, and merges the two. After that it downloads the weights, builds the model, loads the weights into it, and switches the model to evaluation mode.

The diagram below shows that sequence. Read it as a dependency chain rather than a list. Each step consumes something produced by the step before it. The configuration must exist before the model can be constructed, and the download must complete before the loading step can run. The loading step is marked because it is where a mismatch between the downloaded weights and the model's shape first appears.

BASE_CONFIG holds the settings that describe the model in general, and these are shared by every GPT-2 size. It contains four settings. The vocabulary size is 50257. The context length is 1024. The dropout rate is 0.0. The query-key-value bias is True. Together these describe the general shape of the model that the script builds.

The size-specific settings are different. Each GPT-2 size has its own embedding dimension, number of layers, and number of attention heads. These are the values that change when you pick a different size. The base settings do not change between sizes, which is why they are kept in one shared dictionary instead of being repeated for each size.

Two of the four base settings are worth understanding in more detail, because they describe behavior inside the model rather than a simple count.

The script chooses one size and merges its settings into BASE_CONFIG with the dictionary update operation. The size-specific settings are looked up by the chosen key in model_configs. After the merge, BASE_CONFIG contains the four shared settings together with the size's embedding dimension, number of layers, and number of attention heads. The model is then built from this single combined configuration.

The diagram below shows the result of the merge. The four base settings come from BASE_CONFIG, and the three size-specific settings come from the chosen entry in model_configs. Under dictionary update, a key present in both places takes the value from the size entry, while a key present only in the base stays as it was.

Calling the download helper does more than return numbers. Running it causes several files to be downloaded, and each file is listed with its size and transfer progress in the output. The files are not all the same kind of thing, so it helps to group them by role. The grouping below is a way to organize the files. The source lists the files but does not explain each one in detail, so treat the groups as an aid to reading, not a complete specification.

The checkpoint, index, data, and meta files together make up the model's saved checkpoint. The checkpoint data file is the largest of the downloaded files. The encoder.json and vocab.bpe files relate to the vocabulary, which connects text to the numbers a model works with. The hparams.json file holds hyperparameters. The diagram below shows how these groups relate to the download helper and to the two values it returns.

The download helper returns two values, called settings and params. The model is constructed from BASE_CONFIG. Then the function load_weights_into_gpt places the downloaded params into that model, and the loader is described as loading the weights into a GPTModel instance. The last step, model.eval(), prepares the loaded model for use.

Notice that settings is returned but is not passed into the model in the script. The model's shape comes from BASE_CONFIG, while the weights come from params. Keeping these two inputs separate in your head matters. A mismatch between the configuration and the weights does not show up when the configuration is written. It shows up at the loading step, when the weights are placed into the model.

Changing the model size is the most common place where the configuration and the weights drift apart. The following mistakes are easy to make because each one looks reasonable on its own.

The method for debugging this script is the same for any variable. For each variable, write down what you expected it to hold. Then find the line where it was last assigned. The first line whose result differs from your expectation is where the divergence began. Downstream symptoms, such as a model with the wrong shape, often appear far from that line.