Initializing Larger GPT Models

The GPTModel class builds several GPT-2 sizes, and only the configuration file needs to change between them.

Programming

Same Class, Different Sizes

Earlier work set up a 124-million-parameter GPT model, the size known as GPT-2 small. A natural next question follows: once that model works, how much effort does it take to build a larger one? The surprising answer is very little. The larger GPT-2 models in this exercise do not need a new design. They reuse the same GPTModel class, and the difference lies in a configuration file.

This article follows one idea through the exercise. You will watch what happens when the configuration changes, see how the three values that define each GPT-2 size line up across medium, large, and XL, and learn how the total parameter count is connected to those same values. Keep the GPT-2 small model in mind as a reference point throughout, because it is the one size whose total parameter count the source gives directly.

The GPTModel Template

A useful way to think about the GPTModel class is as a template. The template describes how a GPT model is assembled: it creates embeddings, stacks transformer blocks, and includes multi-head attention. The template itself does not contain the specific numbers that make one model different from another. The configuration supplies those numbers and fills in the template. Change the numbers, and you change the size of the model without rewriting the template.

Which config values change between GPT-2 sizes, and which parts of the GPTModel code stay the same? The class stays fixed; the configuration supplies the values it reads.

The diagram shows the key design choice. The GPTModel class sits in the middle as fixed code. The configuration file sets three values, and the class reads them. Because the class takes these values from the configuration rather than having them written into its code, a single class can represent several different sizes. Changing the configuration changes the model, and changing the class does not need to happen at all for this exercise.

The Three Configuration Values

Definition

Embedding dimension: the number of dimensions in the embeddings, which sets how wide the vector representing each token is.

Definition

Transformer block count: the number of transformer blocks, which sets how many stacked processing stages the model has.

Definition

Attention head count: the number of multi-head attention heads, which sets how many attention mechanisms run in parallel inside each block.

Each larger GPT-2 variant is described by these three numbers. Together they define the architecture. The exercise names exactly these three values, and each larger size raises all of them at once. The model is not built by adding one new kind of part. It is built by making the same kinds of parts wider and more numerous.

Tracing the Switch from Small to Medium

Follow the change step by step. Start with the GPT-2 small configuration, which defines the 124-million-parameter model. Your goal is to produce GPT-2 medium while keeping the GPTModel code exactly as it is.

Worked example

Building GPT-2 Medium from the Small Configuration

Produce a GPT-2 medium model by changing configuration values only, without rewriting the GPTModel class.

1
Start with the small configuration
The configuration holds the three values that define GPT-2 small. The class is already known to work, since it built the 124-million-parameter model.
2
Widen the embeddings
Replace the embedding dimension with 1,024. Each token is now represented by a wider vector.
3
Deepen the stack
Replace the transformer block count with 24. The model now stacks 24 processing stages on top of one another.
4
Split attention across more heads
Replace the attention head count with 16. Sixteen attention mechanisms now run in parallel inside each block.
5
Pass the new configuration to the same class
The GPTModel class reads the new values and builds a model with the same structure at a larger scale. No line of its code changes.

Result

The same GPTModel class now builds GPT-2 medium, with 1,024-dimensional embeddings, 24 transformer blocks, and 16 attention heads.

Comparing the Three Larger Sizes

The table collects the values stated in the source. GPT-2 small appears as the baseline with its known size of 124 million parameters. The source does not list its three configuration values in this passage, so those cells are left open rather than guessed.

Size	Embedding dimension	Transformer blocks	Attention heads	Total parameters
GPT-2 small	not stated in source	not stated in source	not stated in source	124 million
GPT-2 medium	1,024	24	16	work out from configuration
GPT-2 large	1,280	36	20	work out from configuration
GPT-2 XL	1,600	48	25	work out from configuration

The three configuration values that define each GPT-2 size, with the known GPT-2 small total as the reference point

How do the three GPT-2 sizes differ in embedding dimension, block count, and head count? Each column is one size, and every size raises all three values together.

Read the columns from top to bottom and the pattern is clear. Each larger model increases all three values, but not by the same proportion. Going from medium to large to XL, the embedding dimension rises from 1,024 to 1,280 to 1,600, the block count from 24 to 36 to 48, and the head count from 16 to 20 to 25. Reading the values across the rows shows which dimension changes most in relative terms, and that comparison is a useful check when you read a configuration.

Reading the Configuration Dictionary

In practice, the three values are stored in a configuration that the GPTModel class reads. The snippet below is an illustrative sketch. The key names are not quoted from the source, so use whatever names your course's configuration actually defines. The point is that the size of the model lives in these values, not in the class code.

python

# Illustrative configuration for GPT-2 medium
# Only these values differ between sizes; the GPTModel code is unchanged
gpt2_medium_cfg = {
    'emb_dim': 1024,   # embedding dimension: width of each token vector
    'n_layers': 24,    # number of transformer blocks
    'n_heads': 16      # number of multi-head attention heads
}

# The same class receives a different dictionary for each size
# (medium, large, or XL) and builds a model at that scale

# Illustrative configuration for GPT-2 medium
# Only these values differ between sizes; the GPTModel code is unchanged
gpt2_medium_cfg = {
    'emb_dim': 1024,   # embedding dimension: width of each token vector
    'n_layers': 24,    # number of transformer blocks
    'n_heads': 16      # number of multi-head attention heads
}

# The same class receives a different dictionary for each size
# (medium, large, or XL) and builds a model at that scale

Notes on the code

emb_dim sets the width of the vector for each token
n_layers sets how many transformer blocks are stacked
n_heads sets how many attention heads run in parallel

Counting Total Parameters

The bonus task asks for the total number of parameters in each model. A parameter is a learnable number inside the model, and the total is the count of all of them. The total is not stored as a separate setting. It is determined by the architecture the configuration describes. Since the embedding dimension, the number of transformer blocks, and the number of attention heads all define that architecture, they are the inputs to the count.

How do the configuration values combine, step by step, into the total parameter count? Each step identifies, counts, or sums one part of the model.

1
Identify the components that the configuration controls. The template creates embeddings, stacks transformer blocks, and includes multi-head attention, so these are the parts whose size depends on the three values.
2
Count the parameters that each component contributes, using the configuration values for that size.
3
Add the component counts together to get the total for that model.
4
Verify the result against a model you have actually built. The GPT-2 small total of 124 million is the anchor for checking your method before you apply it to larger sizes.

Checking Results Against GPT-2 Small

Before you calculate, make a prediction. The source gives one solid expectation: a larger model raises all three configuration values, so its total should be larger too. Think about the order of the three sizes before you read the answer.

What do you think happens?

Going from GPT-2 medium to GPT-2 large to GPT-2 XL, should the total parameter count rise, fall, or stay flat?

It rises across the three sizes
It falls across the three sizes
It stays the same across the three sizes

Reveal answer

Answer:

It rises across the three sizes.

Each step up raises the embedding dimension, the block count, and the head count together. Since the parameter total is determined by these values, the total grows in the same order. If your computed totals do not increase from medium to large to XL, recheck the inputs you used.

The source does not supply the totals for GPT-2 medium, large, and XL. Those are the calculation you are asked to perform. Use the 124-million figure for GPT-2 small as your sanity check: apply your counting procedure to the small configuration first, confirm that it lands near 124 million, and only then apply the same procedure to the larger configurations.

Mistakes When Changing Configuration

Common mistake
Rewriting the GPTModel class to get a bigger model
Why it fails
The class is the template and stays the same across sizes. The size comes from the configuration values.
Do this instead
Change the configuration values and pass them to the same class.
Common mistake
Changing only one of the three values
What it looks like
Raising the embedding dimension to 1,024 while leaving the block count at the small-model value
Why it fails
Each larger size raises all three values together. A partial change does not reproduce the target size.
Do this instead
Update all three values for the size you want, then check the result.
Common mistake
Assuming a larger value always means a proportionally larger model in every dimension
Why it fails
The source notes that each size raises the three settings but not by the same proportion.
Do this instead
Compare the actual values across the table instead of assuming equal growth.
Common mistake
Expecting the total to be stored as a setting
Why it fails
The total parameter count is determined by the architecture, not written into the configuration.
Do this instead
Calculate the total from the component counts and check it against the known 124-million GPT-2 small size.
Common mistake
Choosing head counts that do not split the embedding evenly
Why it fails
Each head works on its own slice of the embedding vector, so the dimension must divide evenly across heads.
Do this instead
Check that the embedding dimension divides by the head count, as it does for the three GPT-2 sizes in this exercise.

Practice Reading the Sizes

EASY

Using the comparison table, identify which GPT-2 size has 48 transformer blocks and 25 attention heads. Then state the embedding dimension for that size, and explain in one sentence why this model is not built from a different class than GPT-2 medium.

Hints

Look at the rows for GPT-2 large and GPT-2 XL, and find the one with 48 blocks.
The embedding dimension is in the same row as the block count and head count.
Think about what the template is versus what the configuration supplies.

Your attempt (not saved)

MEDIUM

Starting from the GPT-2 medium configuration, write out the three changes needed to produce GPT-2 large. Then state whether any line of the GPTModel class must change, and justify your answer using the template idea.

Hints

List the new values for the embedding dimension, block count, and head count.
Ask whether the class is the template or the configuration.

Your attempt (not saved)

Key Takeaways

The GPTModel class works as a fixed template, and the configuration file supplies the values that set the size of the model.
GPT-2 medium uses 1,024 embedding dimensions, 24 transformer blocks, and 16 attention heads; GPT-2 large uses 1,280, 36, and 20; GPT-2 XL uses 1,600, 48, and 25.
Each larger size raises all three configuration values together, so one class can produce several sizes without any code change.
The total parameter count is determined by the architecture the configuration describes. Calculate it by identifying, counting, and summing each component, then check the method against the known 124-million GPT-2 small size.
Totals should rise from medium to large to XL, and a configuration whose totals do not increase in that order should be rechecked.

Related concepts

Multi-Head Attention and Parallel Heads

More concepts starting with I

Same Class, Different Sizes

The GPTModel Template

The Three Configuration Values

Tracing the Switch from Small to Medium

Comparing the Three Larger Sizes

Reading the Configuration Dictionary

Counting Total Parameters

Checking Results Against GPT-2 Small

Mistakes When Changing Configuration

Practice Reading the Sizes

Key Takeaways

Related Topics

Related concepts