Evaluating a Loaded GPT Model

A GPT model that has been loaded with pretrained weights is not finished until you have checked that it runs and produces the output you expect. One of the simplest checks is to ask the model to continue a short piece of text and then examine what comes back. This article follows one such run from start to finish. The program is executed, its printed result is shown, and then each part of that result is traced back to the code that produced it.

Before you read on, try to predict something concrete. The generation call receives four input tokens and is asked to generate six new tokens. How long will the returned sequence of output token IDs be?

The model's settings begin in a dictionary named BASE_CONFIG. This dictionary holds the four settings that describe the GPT-2 model and that are shared by every GPT-2 size. The vocab_size entry is 50257. The context_length entry is 1024. The drop_rate entry is the dropout rate, and it is set to 0.0 here. The qkv_bias entry is True, which sets the query-key-value bias. Because these four values do not change from one size to another, they live in a base dictionary that every size can use.

The four base settings do not describe a particular size. The size-specific values are stored separately, in a second dictionary named model_configs. Each key in model_configs is a model name such as gpt2-medium (355M), and each value gives the emb_dim, n_layers, and n_heads for that model. The line BASE_CONFIG.update(model_configs[CHOOSE_MODEL]) copies the three size-specific entries of the chosen model into BASE_CONFIG. After that line runs, one dictionary holds both the shared settings and the settings for the chosen size.

The download step needs a short model size string, but the CHOOSE_MODEL variable holds a longer name that includes the parameter count in parentheses. The source derives model_size from CHOOSE_MODEL with one chained expression, which splits the name on spaces and strips the parentheses. For the medium model, this produces 355M.

The value 355M is then passed to download_and_load_gpt2 as model_size, together with models_dir set to gpt2. That call returns two values, named settings and params. The weight-loading step that follows uses params, which holds the pretrained parameters.

Loading pretrained GPT-2 weights is a two-part job. A GPT model is built from a set of settings, and the pretrained weights are stored values that were produced for a specific model shape. First you create a GPT model object with the right settings, and then you pass the pretrained parameters into that object. The listing does this in two lines. The first line, model = GPTModel(BASE_CONFIG), creates a GPT model instance from the merged configuration. The second line, load_weights_into_gpt(model, params), takes that instance and the pretrained parameters and loads the weights into the model.

The order of these lines matters. The model must exist before the second line runs, because load_weights_into_gpt receives the model object as its first argument. The configuration and the downloaded parameters are two separate inputs. The model is created from the configuration alone, and the loading function is the point where the two inputs meet.

The same model object passes through three states in the listing. It starts as a GPT model built from BASE_CONFIG. The loading call changes it so that it holds the pretrained weights. The final line, model.eval(), is the last statement in the listing. The source does not explain what that line does in the loading listing, so this article covers evaluation mode separately in the next section.

The generation step is a single call to generate_text_simple. The function receives four things. It receives the model. It receives an encoded input tensor named idx, which holds the starting tokens. It receives a limit of 6 new tokens through max_new_tokens. It receives a context size, taken from the model configuration as its context length. Each argument answers a separate question: which model to use, what the starting tokens are, how far to continue, and how much preceding text the model may look at.

The function returns output token IDs, not readable words. The reported output length of 10 equals the 4 input tokens plus the 6 newly generated tokens. Each step of the run adds one token to the sequence, so the length grows from 4 to 10 over six steps.

Before generating anything, the model is placed into evaluation mode by calling model.eval(). The source describes this as the switch that disables random components such as dropout. Dropout is a training-time mechanism, so it is only used while the model is learning. When the model is only being run to produce output, that randomness is not wanted, and evaluation mode turns it off.

When a loaded model does not match what you expected, the fastest route is to ask which line of the listing first produced a value you did not expect. Each line depends on the ones above it, so the earliest wrong value usually points to the real cause. The mistakes below follow the order of the listing.

Suppose a run returns a sequence of 7 token IDs, and the input had 4 tokens with max_new_tokens set to 6. Use the output length as a checkpoint to decide what should be investigated first.