Initializing the 124-Million-Parameter GPT Model
The GPTModel class does not have its sizes written into its code. It reads them from the configuration. Changing the configuration changes the model, and for this exercise the class itself does not need to be rewritten.
One Class, Several GPT-2 Sizes
Earlier work built a GPT model with about 124 million parameters. That model is known as GPT-2 small. This article asks a related question: if you want a bigger GPT-2 model, do you have to write new model code? The answer is no. The same GPTModel class can produce several different GPT-2 sizes, and what changes between them is the configuration file.
The GPTModel class does not have its sizes written into its code. It reads them from the configuration. Changing the configuration changes the model, and for this exercise the class itself does not need to be rewritten.
The Three Values That Define a GPT Model
A GPT model is described by a small set of configuration values. The exercise names three of them. The first is the number of dimensions in the embeddings, which sets how wide the vector representing each token is. The second is the number of transformer blocks, which sets how many stacked processing stages the model has. The third is the number of multi-head attention heads, which sets how many attention mechanisms run in parallel inside each block.
Because all three values are read from the configuration, each one can be changed independently in the file. The table of GPT-2 sizes later in this article shows how they are set for each variant.
Inside the GPTModel Structure
The GPTModel contains embeddings and a stack of transformer blocks. Each transformer block contains multi-head attention. The number of transformer blocks sets how many times that block-level structure is stacked, and the number of heads sets how many attention mechanisms run side by side inside each block.
The model is not built by adding a new kind of part as it grows. It is built by making the same kinds of parts wider and more numerous. A bigger GPT-2 variant uses the same pieces as the smaller one, arranged at a larger scale.
Comparing the GPT-2 Configurations
The four GPT-2 variants share one architecture and differ only in the three configuration values. The table below collects the values stated in the source. GPT-2 small is included as the baseline with its size of 124 million parameters, but the passage does not list its three configuration values, so those cells are marked as not listed.
| Model | Embedding dimensions | Transformer blocks | Attention heads | Parameters |
|---|---|---|---|---|
| GPT-2 small | Not listed in this passage | Not listed in this passage | Not listed in this passage | 124 million |
| GPT-2 medium | 1,024 | 24 | 16 | Not given in this passage |
| GPT-2 large | 1,280 | 36 | 20 | Not given in this passage |
| GPT-2 XL | 1,600 | 48 | 25 | Not given in this passage |
GPT-2 variants and their configuration values, as stated in the source
Read the rows from top to bottom. Each larger model raises all three values at once. Medium is wider, deeper, and has more heads than small would need to be at its own scale, and large and XL continue the same pattern.
Switching from Small to Medium
Building GPT-2 medium from the GPT-2 small configuration
Starting from the configuration that defines GPT-2 small, produce GPT-2 medium without rewriting the model code.
- 1Start with the small configuration
Begin with the three values that define the small model. The passage does not list them, so you would read them from your own configuration file.
- 2Replace the embedding dimensions
Widen the embeddings to 1,024 dimensions, so each token is represented by a wider vector.
- 3Replace the number of transformer blocks
Deepen the stack to 24 transformer blocks.
- 4Replace the number of attention heads
Split attention across 16 heads inside each block.
- 5Build the model
The GPTModel class takes the new configuration and builds a model with the same structure at a larger scale. Nothing else is rewritten.
The same class now produces GPT-2 medium, with 1,024 embedding dimensions, 24 transformer blocks, and 16 attention heads.
Connecting Configuration Values to the Parameter Count
A parameter is a learnable number inside the model. The total parameter count is not stored as a separate setting. It is determined by the architecture that the configuration describes. The embedding dimensions, the number of transformer blocks, and the number of attention heads all define that architecture, so they are the inputs to the count.
The source gives one anchor value: GPT-2 small has 124 million parameters. You can use that model as a reference point when checking your work on larger models. The passage does not provide totals for GPT-2 medium, large, or XL. Those totals come from calculating with the configuration values, not from looking them up in the text.
Predicting the Trend Across Sizes
What do you think happens?
As you move from GPT-2 medium to GPT-2 large to GPT-2 XL, what should happen to the total parameter counts?
Reveal answer
They should rise across those three models.
Going from medium to large to XL raises all three inputs (embedding dimensions, transformer blocks, and attention heads), so the totals should increase in that order. If your computed totals do not increase this way, recheck the inputs you used.
Mistakes When Reading or Changing Configurations
Common mistake Rewriting the GPTModel class to get a larger model
Why it failsThe class reads its sizes from the configuration. For this exercise, changing the configuration is what changes the model, and the class does not need to be edited.
Do this insteadChange the configuration values and let the same class build the new model.
Common mistake Changing only one of the three values when moving to a different GPT-2 size
What it looks likeKeeping the small model's 12 blocks while setting the medium embedding dimension and head count.
Why it failsEach larger variant raises all three values together. Mixing values from different sizes does not match any of the variants in the table.
Do this insteadReplace all three values with the set that belongs to the target model.
Common mistake Treating the 124 million figure as the size of every model
Why it failsThe 124 million figure is the anchor for GPT-2 small only. The passage gives no totals for medium, large, or XL.
Do this insteadUse 124 million as a reference point and calculate the other totals from their configuration values.
Common mistake Expecting the parameter count to fall as the configuration grows
Why it failsLarger configuration values define a larger architecture, so the parameter count grows with them.
Do this insteadCheck that totals increase from medium to large to XL, and recheck inputs if they do not.
Common mistake Looking for stored parameter totals in the configuration
Why it failsThe total is not a separate setting. It follows from the architecture the configuration describes.
Do this insteadCompute the total from the embedding dimensions, transformer blocks, and attention heads.
Checking Your Understanding of the Sizes
Using the table of GPT-2 medium, large, and XL values, identify which of the three values stays the same and which changes between each consecutive pair of sizes. Then explain in one or two sentences why the total parameter count should rise across the three models, even though the passage does not give those totals.
Hints
- Compare the medium row with the large row, then the large row with the XL row.
- Every consecutive pair raises all three values, so none of the three stays fixed.
- The parameter count depends on the architecture the three values define, so larger inputs produce a larger architecture.
Key Takeaways
- The GPTModel class builds several GPT-2 sizes, and only the configuration file needs to change between them.
- Three configuration values define the architecture: embedding dimensions, transformer blocks, and multi-head attention heads.
- GPT-2 medium uses 1,024 dimensions, 24 blocks, and 16 heads; large uses 1,280, 36, and 20; XL uses 1,600, 48, and 25.
- The total parameter count is determined by the architecture these values describe, with GPT-2 small at 124 million as the only anchor given in the passage.