Converting GPT Model Outputs into Text

Initializing the 124M GPT model and printing its output tensor gives a shape of [2, 4, 50257], where 50257 is the tokenizer's vocabulary size.

Programming

A Model Named 124M With a Bigger Count

When you initialize the 124M GPT model from the GPT_CONFIG_124M configuration and add up the sizes of its parameter tensors, the total comes out noticeably larger than 124 million. Many learners assume something has gone wrong, either a bug in the setup or an error in the counting. Neither is true. The gap comes from a design choice in the original GPT-2 architecture, and understanding that choice is the main purpose of this article.

This article follows a simple loop. First you predict a number, then you trace how the parameters are counted, then you learn the design idea that explains the difference, and finally you read the output tensor that the initialized model produces. Each step builds on the one before it.

Counting the Parameters Step by Step

The total is computed by adding the element count of every parameter tensor in the model. Each tensor's element count is given by its numel() method, and the model's parameters are reached through model.parameters(). Summing those counts is written with a generator expression, as in the code below.

python

total_params = sum(p.numel() for p in model.parameters())
print(f'Total number of parameters: {total_params:,}')

total_params = sum(p.numel() for p in model.parameters())
print(f'Total number of parameters: {total_params:,}')

What do you think happens?

Before you look at the result, commit to a number. What total do you expect this sum to produce for the initialized 124M model?

Reveal answer

Answer:

163,009,536

The raw total computed with numel() is 163,009,536. This is larger than the 124 million name because the output layer is counted separately in this implementation, even though its weights are closely related to the token embedding layer.

So the sum over all parameter tensors gives 163,009,536. That is the raw count of the model object as built in this implementation. The next question is which part of that total accounts for the gap between this number and the advertised 124 million.

Weight Tying Explained

The resolution to the mismatch is weight tying. In the original GPT-2 architecture, the weights of the token embedding layer are reused in the output layer. Instead of storing two separate matrices, the model keeps one set of values, and both layers draw on that same set.

Definition

Weight tying means reusing the token embedding weights in the output layer. The token embedding layer and the output layer share one weight matrix rather than each storing its own copy.

Under GPT-2 weight tying, which two layers point to the same weight matrix, and how does that one matrix serve both the token-embedding input step and the output-projection step?

The diagram shows the key idea: both layers point to one stored matrix. The output layer does not add its own large block of values. It reads from the same stored weights that the embedding layer uses.

Subtracting the Output Layer's Parameters

The source computes the tied count by taking the full total and removing the output layer's parameters. The output layer alone accounts for 38,597,376 values here. Because that block is redundant under weight tying, the trainable count is the full count minus this block.

When the output layer reuses the embedding weights, which parameters are counted once instead of twice, and how does the total change?

Formula, step by step

1
Start with the raw total from summing numel() over all parameter tensors: 163,009,536.
2
Identify the output layer's parameters, which under weight tying are reused from the embedding weights: 38,597,376.
3
Subtract the output layer's parameters from the raw total to get the trainable count: 124,412,160.
4
The result is the 124 million figure in the model's name, and the arithmetic checks out exactly.

Worked example

Tracing the tied count

The initialized 124M model has a raw parameter total of 163,009,536. The output layer holds 38,597,376 parameters that weight tying makes unnecessary. What is the trainable count with weight tying?

1
Confirm the raw total
Summing p.numel() over model.parameters() gives 163,009,536. This counts the output layer separately, as this implementation does.
2
Identify the redundant block
Under weight tying, the output layer reads from the embedding weights. Its 38,597,376 values are therefore not separate trainable parameters.
3
Subtract
163,009,536 minus 38,597,376 leaves 124,412,160.

Result

The trainable count with weight tying is 124,412,160, which matches the 124 million name once rounded.

Reading the Output Tensor Shape

Parameter counts describe the model's size, but initialization also produces an output you should be able to read. The model is run on a batch of two input texts, each with four tokens. The output tensor therefore has shape [2, 4, 50257]. Each of the three dimensions has a clear meaning.

What does each dimension of the output tensor index (batch, token position, vocabulary entry), and where does 50257 come from?

The first dimension is the batch size, which is 2 here. The second is the sequence length, which is 4 tokens per text. The last dimension is the vocabulary size of the tokenizer, 50257. Each token position therefore receives a vector of 50,257 numbers, one for each token in the vocabulary.

Common Mistakes When Counting Parameters

Common mistake
Assuming the parameter sum is wrong because it does not equal the model's name
What it looks like
Summing numel() over the model gives 163,009,536, and someone concludes the build has a bug.
Why it fails
The name 124M describes the GPT-2 design including weight sharing. The raw count of the built object can differ from that name.
Do this instead
Check whether the difference is explained by weight tying before suspecting an error.
Common mistake
Subtracting the wrong block when applying weight tying
What it looks like
Removing the token embedding parameters instead of the output layer's parameters.
Why it fails
The source subtracts the output layer's parameters, because the output layer is the one that reuses the embedding weights.
Do this instead
```
Subtract 38,597,376 from 163,009,536 to get 124,412,160.
```
Common mistake
Treating weight tying as a rule every GPT model must follow
What it looks like
Assuming the implementation in this chapter should share weights because the GPT-2 design does.
Why it fails
The source presents weight tying as a trade-off. This implementation keeps the layers separate, and modern LLMs also use separate layers.
Do this instead
Understand weight tying as an option with memory and compute benefits, and note the trade-off in training behavior the author reports.
Common mistake
Reading the last dimension of the output as a list of words
What it looks like
Interpreting each 50257-value vector as a single token or word.
Why it fails
Each vector holds one number per vocabulary entry, and these numbers are not yet converted into tokens.
Do this instead
Read the final dimension as the vocabulary size, and wait for the later conversion step to map vectors to tokens.

Practice: Check Your Count

EASY

A model's raw parameter total is 163,009,536. Under weight tying, the output layer's 38,597,376 parameters are reused from the token embedding weights and are not counted again. Calculate the trainable count with weight tying, and then state what the output tensor shape [2, 4, 50257] tells you about the batch, the sequence length, and the vocabulary.

Hints

Subtract the output layer's parameters from the raw total.
The first dimension of the output tensor is the batch size, the second is the sequence length, and the last is the vocabulary size.

Your attempt (not saved)

Key Takeaways

1
The raw parameter total of the initialized 124M model is 163,009,536, which is larger than the 124 million name because the output layer is counted separately in this implementation.
2
Weight tying reuses the token embedding weights in the output layer, so the output layer's 38,597,376 parameters are not counted again.
3
Subtracting those parameters gives the trainable count of 124,412,160, which is the figure the 124M name reflects.
4
The output tensor has shape [2, 4, 50257]: batch size 2, sequence length 4, and vocabulary size 50257 from the tokenizer.
5
Weight tying is a trade-off that reduces memory and computation, while separate layers were kept in this implementation because the author found they train and perform better.

At a glance

The raw parameter total of the initialized 124M model is 163,009,536, larger than the 124 million name because the output layer is counted separately in this implementation.
Weight tying reuses the token embedding weights in the output layer, so the output layer's 38,597,376 parameters are not counted again.
Subtracting those parameters from the raw total gives the trainable count of 124,412,160, which matches the 124M name.
The output tensor shape [2, 4, 50257] means batch size 2, four token positions per text, and a vocabulary of 50257 entries from the tokenizer.

Related concepts

More concepts starting with C