GPT Model Configuration

A working listing for loading a pretrained GPT model does three things in a single run. It downloads GPT-2 weights, builds a GPTModel object, and loads those weights into that object. Before looking at any theory, read the listing once and notice that it has three distinct stages: a configuration stage, a download stage, and a construction-and-loading stage. Each stage produces something that a later stage needs.

As you work through this article, keep one question in mind for every stage: what value does this stage produce, and which later line depends on it? This question is also the most useful habit when something goes wrong. A failure almost always appears at the first line whose expected input was not produced correctly.

The configuration for a GPT model is held in a dictionary named BASE_CONFIG. Its contents describe the shape and behavior of the model that GPTModel will construct. The four base settings written directly in the listing are vocab_size set to 50257, context_length set to 1024, drop_rate set to 0.0, and qkv_bias set to True.

The line BASE_CONFIG.update(model_configs[CHOOSE_MODEL]) does two things in sequence. First, the expression model_configs[CHOOSE_MODEL] uses the value stored in CHOOSE_MODEL as a key and retrieves the matching size dictionary. Second, update copies the entries of that size dictionary into BASE_CONFIG. The variable CHOOSE_MODEL is therefore the single switch that decides which architecture the rest of the listing will build.

The model_configs dictionary holds four GPT-2 sizes. Each size has its own embedding dimension, number of layers, and number of heads. The gpt2-small model, for example, has an embedding dimension of 768, 12 layers, and 12 heads. The source lists all four entries, and the table in the next part of this article shows how the three values for each size are arranged.

Every step up in size changes all three values together. Larger models have larger embedding dimensions, more layers, and more heads. Because all three values are merged into BASE_CONFIG by the same update call, switching CHOOSE_MODEL changes the whole architecture at once rather than one setting at a time.

The download function needs a short size label, not the full name with its parentheses. The line that creates that label is a chain of three string operations, and each one can be checked on its own: split on a space, take the last piece, strip the opening parenthesis, and strip the closing parenthesis.

The label is then passed to the download call, download_and_load_gpt2(model_size=model_size, models_dir="gpt2"). This function comes from the gpt_download module. It returns two values, which the listing unpacks into the names settings and params. The model_size argument selects which GPT-2 model to fetch, and models_dir names the directory used for it. The source lists these arguments but does not describe how files are stored internally, so this article treats the function as a black box that returns settings and params.

The listing runs in three stages. First the model is configured, which means BASE_CONFIG is built from the base settings and the chosen size entry. Second, the settings and parameters are downloaded. Third, the model is constructed and the weights are loaded. Two independent paths feed the final step. One path starts from the chosen name and ends in a configuration that defines the model's shape. The other path starts from the same name, passes through the download function, and ends in the weights. Both paths must be complete before the load can run.

The GPTModel class comes from chapter04. It receives the configuration dictionary, which at this point already contains both the base settings and the size-specific entries. The next line takes the instance and the downloaded parameters and loads the pretrained weights into that instance. The source identifies this function as the mechanism for placing pretrained weights into a GPTModel instance. GPTModel(BASE_CONFIG) must run before load_weights_into_gpt(model, params), because the second call needs the model instance that the first call creates.

Because each stage produces a value that a later stage consumes, a failure usually surfaces at the later line that first expects the missing or incorrect value. The diagram below traces the flow of values between the actors in the listing. If the configuration is wrong, the error is caused by the configuration stage but appears when the model is constructed. If the parameters are missing or do not fit the model, the cause lies in the download stage but the error appears at the load call.