GPT Model Architecture and Its Configuration
BASE_CONFIG holds the shared GPT-2 settings, and BASE_CONFIG.update copies the chosen size's emb_dim, n_layers, and n_heads into it.
A Model Built in Two Stages
A GPT model is built from a set of settings, and pretrained GPT-2 weights are a set of stored values that were produced for one specific model shape. Loading those weights into a GPT model is therefore a two-part job. First you create a GPT model object with the right settings. Then you pass the pretrained parameters into that object. This article follows one complete listing that does exactly that, and it treats the listing as a sequence of steps where each line depends on the values produced above it.
Keep the listing in mind as you read. Before the article works through each line, it helps to see the overall shape: a configuration dictionary is assembled, a model is created from it, and downloaded parameters are loaded into that model. Each of those ideas gets its own section below.
Predicting BASE_CONFIG After the Update
The BASE_CONFIG dictionary holds four settings that describe the GPT-2 model. The vocab_size entry is 50257 and the context_length entry is 1024. The drop_rate entry is the dropout rate, set to 0.0 here. The qkv_bias entry is True, which sets the query-key-value bias. These four values are the same regardless of which GPT-2 size is chosen, so they live in a base dictionary that every size shares.
vocab_size: 50257 is the value the source gives for this shared settingcontext_length: 1024 is the value the source gives for this shared settingdrop_rate: 0.0 is the dropout rate used hereqkv_bias: True sets the query-key-value bias
The four base settings do not describe a size. The size-specific values come from a second dictionary named model_configs. Each key in model_configs is a model name such as gpt2-medium (355M), and each value gives emb_dim, n_layers, and n_heads for that model. The line BASE_CONFIG.update(model_configs[CHOOSE_MODEL]) copies the chosen size's three entries into BASE_CONFIG. Afterwards, one dictionary holds both the shared settings and the size-specific settings.
model_configs[CHOOSE_MODEL] looks up the size entry for the chosen model name- update copies that entry's keys into BASE_CONFIG
What do you think happens?
After BASE_CONFIG.update(model_configs[CHOOSE_MODEL]) runs, which keys does BASE_CONFIG hold, and how many are there?
Reveal answer
The four base keys plus emb_dim, n_layers, and n_heads, for seven keys in total.
The base dictionary keeps its four entries, and update adds the three size entries from the chosen model. The two sets of keys do not overlap, so nothing is replaced and the total is seven.
Read the diagram column by column. The left column is BASE_CONFIG before the update line: four entries with fixed values. The right column is the same dictionary afterwards. The four shared entries carry over unchanged. The three highlighted keys, emb_dim, n_layers, and n_heads, are the ones copied in from the chosen size. None of the original four keys is removed or overwritten, which is why the answer to the prediction is seven keys.
Deriving the Short Model Size
The download step needs a short model size string, but CHOOSE_MODEL holds a longer name that includes the parameter count in parentheses. The source derives model_size from CHOOSE_MODEL by splitting on spaces and stripping parentheses, which produces 355M for the medium model. The source uses one chained expression for this. The worked example below traces the same operations one at a time on the medium model name.
Turning the medium model name into 355M
CHOOSE_MODEL is the name gpt2-medium (355M). Produce the short model_size string that the download step needs.
- 1Start with the full name
CHOOSE_MODEL holds gpt2-medium (355M), with a space between the model name and the parenthesized parameter count.
- 2Split on spaces
Splitting on the space separates the name into two pieces: gpt2-medium and (355M).
- 3Keep the parameter count piece
The piece that carries the size is (355M), the part after the space.
- 4Strip the parentheses
Removing the opening and closing parentheses leaves 355M.
model_size is 355M.
model_sizeis the short string 355M derived from CHOOSE_MODELmodels_diris set to gpt2- The call returns two values, settings and params. This article focuses on params.
Building the Model and Loading Its Weights
The loading itself takes two lines. The first creates a GPT model instance from the merged configuration. The second takes that instance and the pretrained parameters and loads the weights into the model. The model must exist before the second line runs, because load_weights_into_gpt receives the model object as its first argument.
GPTModel(BASE_CONFIG)creates the model from the merged configurationload_weights_into_gpt(model, params)loads the pretrained parameters into that modelmodel.eval(); is the last statement in the listing; the source does not explain it
The diagram shows two separate inputs. The configuration produces the model on its own, and the downloaded parameters arrive separately. The loading function is the point where the two inputs meet. The same model object passes through three states in the listing. It starts as a GPT model built from BASE_CONFIG. The loading call then changes it so that it holds the pretrained weights. The final statement, model.eval();, comes last, and the source does not explain what it does, so this article does not expand on it.
Where to Look When a Loaded Model Misbehaves
When a loaded model does not match what you expected, the fastest route is to ask which line of the listing first produced a value you did not expect. Each line depends on the ones above it, so the earliest wrong value usually points to the real cause. Work top to bottom: check the configuration dictionary, then the model_size string, then the downloaded params, then the loading call.
Common mistake Inspecting the loading call first when the model shape looks wrong
Why it failsThe model shape comes from BASE_CONFIG, which is built before GPTModel runs. A wrong shape usually traces back to the update line or to the size name used to choose the entry.
Do this insteadPrint BASE_CONFIG after the update line and confirm that emb_dim, n_layers, and n_heads match the chosen size.
Common mistake Assuming the update call replaces the shared settings
What it looks likeExpecting vocab_size or context_length to disappear after the update line runs.
Why it failsThe update copies three size entries into the dictionary. The four base keys are kept, and the two sets of keys do not overlap.
Do this insteadCheck the key list after the update and expect the four base keys plus the three size keys.
Common mistake Passing the model name instead of the short size string
What it looks likePassing gpt2-medium (355M) where the download step needs 355M.
Why it failsThe download step expects the short
model_sizestring that is derived from CHOOSE_MODEL.Do this insteadConfirm that model_size is 355M for the medium model before calling download_and_load_gpt2.
Common mistake Calling load_weights_into_gpt before the GPTModel line has created the model object
Practice: Tracing the Listing
Suppose a learner runs the listing and the loaded model has a different embedding size than the medium model should have. Using only the order of the lines, name the earliest line you would inspect, and explain what you would check on it.
Hints
- Start from the top of the listing, not from the loading call.
- Think about which line first writes
emb_diminto the configuration. - The fix is usually in the line that produced the wrong value, not in the line that displayed it.
Key Takeaways
- BASE_CONFIG holds four shared GPT-2 settings:
vocab_size50257,context_length1024,drop_rate0.0, andqkv_biasTrue. BASE_CONFIG.update(model_configs[CHOOSE_MODEL])adds the chosen size'semb_dim,n_layers, andn_headswithout removing the shared keys.model_sizeis derived from CHOOSE_MODEL by splitting on spaces and stripping parentheses, producing 355M for the medium model.GPTModel(BASE_CONFIG)creates the model, andload_weights_into_gpt(model, params)loads the pretrained parameters into it, in that order.- When a loaded model misbehaves, inspect the listing line by line from the top, because the earliest wrong value usually points to the cause.