Counting Parameters in a GPT Model

The size of a GPT model is its parameter count multiplied by the bytes per parameter, and this example assumes 4 bytes for each 32-bit float.

Programming

Why Model Size Matters

Once a GPT model has been built in code, a natural question follows: how much storage does it take up? Training produces a large collection of learned numbers, and those numbers have to live somewhere, either in memory while the model runs or in a file on disk. Before you can plan hardware or judge whether a model fits on a given device, you need a way to turn the model's parameter count into a size.

This article works through that conversion for a GPT model with 163 million parameters. The goal is not only to arrive at a number but to understand where the number comes from, so that you can repeat the same reasoning for any model whose parameter count you know. The parameter count is the only input that changes from model to model; the method stays the same.

From Parameters to Bytes

Definition

The size of a GPT model is its parameter count multiplied by the bytes per parameter. This example assumes 4 bytes for each parameter, which is the storage used by one 32-bit float.

The calculation can be read as two steps. The first step counts bytes: each parameter contributes a fixed number of bytes, so the total byte count is the parameter count multiplied by that per-parameter size. If we write N for the total number of parameters, the byte count is N times 4.

Rule

Here N is the total number of parameters and the constant 4 is the number of bytes per parameter. Each parameter is one 32-bit float, and a 32-bit float occupies 4 bytes. Because the bytes per parameter is fixed, the byte count depends only on N.

Converting Bytes to Megabytes

The second step changes units. Bytes are divided into larger blocks of 1024 by 1024 to produce megabytes. In this calculation, 1024 times 1024 is how many bytes make up one megabyte. Dividing the byte count by that number gives the size in megabytes.

Rule

Tracing the Size Script

The script below does three things in order: it stores the parameter count, multiplies it by the number of bytes each parameter occupies, and then converts that byte count into megabytes. In the original workflow the parameter count comes from the GPTModel object built earlier. Here the count is written out directly so the arithmetic is easy to follow.

python

total_params = 163000000
bytes_per_param = 4
total_bytes = total_params * bytes_per_param
megabytes = total_bytes / (1024 * 1024)
print(round(megabytes, 2))

total_params = 163000000
bytes_per_param = 4
total_bytes = total_params * bytes_per_param
megabytes = total_bytes / (1024 * 1024)
print(round(megabytes, 2))

Notes on the code

total_params stores the parameter count N
bytes_per_param is 4 because each value is a 32-bit float
total_bytes is the byte count obtained by multiplying N by 4
megabytes divides the byte count by 1024 times 1024
round(..., 2) keeps two decimal places in the printed result

What do you think happens?

What number does the script print?

163.00
621.83
652000000.00
1024.00

Reveal answer

Answer:

621.83

The script computes 163000000 times 4, which is 652000000 bytes. Dividing 652000000 by 1048576 gives approximately 621.83 megabytes. The number 163.00 is the parameter count in millions, not the size, and 652000000 is the byte count before unit conversion.

How do the variables total_params, bytes_per_param, total_bytes and megabytes change in value as each line of the script executes in order?

Worked Example: The 163 Million Parameter Model

Worked example

Storage size of a 163 million parameter GPT model

A GPT model has 163 million parameters, and each parameter is stored as a 32-bit float using 4 bytes. How large is the model in megabytes using the 1024 by 1024 divisor?

1
Write down the parameter count
N = 163000000 parameters.
2
Count the bytes
Multiply N by 4 bytes per parameter: 163000000 times 4 = 652000000 bytes.
3
Compute the divisor
One megabyte in this calculation is 1024 times 1024 = 1048576 bytes.
4
Convert to megabytes
Divide 652000000 by 1048576, which gives approximately 621.83 MB.

Result

The parameters of the 163 million parameter model occupy about 621.83 MB.

How does the parameter count flow through multiplication by 4 bytes per 32-bit float to produce the total byte size of the model?

Scaling and What the Number Leaves Out

Because the bytes per parameter is fixed, size scales linearly with the parameter count. A model with ten times as many parameters, at the same 4 bytes each, needs ten times as many bytes. This proportional relationship is what makes the method reliable: once you know the bytes per parameter, a new parameter count gives a new size with the same two steps.

Reading the Output Tensor Shape

The same section also looks at what GPTModel produces. The model outputs numeric tensors with the shape [batch_size, num_tokens, vocab_size]. Each of the three dimensions answers a different question about the output, and reading them in order tells you how many numbers the model returns for a given input.

The first dimension, batch_size, selects which item in the batch you are looking at. The second dimension, num_tokens, selects the token position within that item. The third dimension, vocab_size, selects one entry in the vocabulary, and the number stored there is the score for that possible next token.

Which index in the output tensor selects the batch item, which selects the token position, and which selects the vocabulary entry holding the score for each possible next token?

python

batch_size, num_tokens, vocab_size = 2, 4, 10
number_of_outputs = batch_size * num_tokens * vocab_size
print(number_of_outputs)

batch_size, num_tokens, vocab_size = 2, 4, 10
number_of_outputs = batch_size * num_tokens * vocab_size
print(number_of_outputs)

Output (expected)

Notes on the code

batch_size = 2 items in the batch
num_tokens = 4 token positions per item
vocab_size = 10 scores per token position
the product counts every number the output returns

The same multiply-then-convert habit used for parameters works for any tensor whose shape you know. The output is a different collection of numbers, but multiplying its dimensions gives its element count, and that count can be turned into bytes exactly as before.

Mistakes With Units and Assumptions

Common mistake
Reporting the parameter count as the model size
What it looks like
Saying the model is 163 MB because it has 163 million parameters.
Why it fails
The parameter count is a number of values, not a number of bytes. Each parameter takes 4 bytes in this calculation.
Do this instead
Multiply the parameter count by the bytes per parameter first, then convert the byte count to megabytes.
Common mistake
Dividing by 1000 instead of 1024 times 1024
What it looks like
Computing 652000000 divided by 1000000 and reporting 652 MB.
Why it fails
This calculation defines one megabyte as 1024 times 1024 bytes, so a different divisor gives a different answer.
Do this instead
Divide by 1024 times 1024 to get 621.83 MB for the 163 million parameter model.
Common mistake
Forgetting the bytes-per-value assumption
Why it fails
The result depends on how many bytes each value uses. A calculation that assumes a different width gives a different size.
Do this instead
Always state the bytes-per-value assumption, here 4 bytes for each 32-bit float, before reporting a result.
Common mistake
Mixing up what the output dimensions mean and reading batch_size as the token count

Practice Estimating a Model's Size

MEDIUM

A different GPT model has 50 million parameters, each stored as a 32-bit float. Estimate its size in megabytes using the 1024 by 1024 divisor, showing the byte count before the conversion.

Hints

Multiply the parameter count by 4 to get the total bytes.
Divide the byte count by 1024 times 1024 to get megabytes.
The answer should be a little under 191 MB.

Your attempt (not saved)

Key Takeaways

A model's size is its parameter count multiplied by the bytes per parameter; this example uses 4 bytes for each 32-bit float.
Bytes become megabytes by dividing by 1024 times 1024, so 163 million parameters give about 621.83 MB.
Size scales linearly with the parameter count once the bytes per parameter are fixed, and the result counts only the learned numbers.
The GPT model's output has shape [batch_size, num_tokens, vocab_size]: the batch index, the token position, and the vocabulary entry holding each next-token score.
Always check the units and the bytes-per-value assumption before reporting a size.

More concepts starting with C