Calculating Model Parameters

A GPT model is defined by its configuration, and every choice in that configuration changes how many learned values the model must store. Before you commit to training a model, it helps to know how large it is. A parameter count tells you roughly how much memory the model will need, and it lets you compare one design against another.

This article approaches the topic from the code outward. You will read a short program that estimates how much memory a model's parameters occupy, and you will practice two skills. The first is counting and comparing the parameters in different modules of a transformer block, such as the feed forward module and the multi-head attention module. The second is converting a total parameter count into a size in bytes and then into megabytes. Both skills depend on one simple chain of arithmetic, so it is worth reading the program line by line before trusting any printed number.

The program has three calculation steps, and each one answers a different question. The first step asks how many bytes the parameters need. It assumes every parameter is a float32 number, which takes 4 bytes, so the byte count is the parameter count multiplied by 4. The second step converts bytes into megabytes. The program divides by 1024 * 1024, which is 1,048,576, rather than by one million. The third step is the print statement, which formats the megabyte value with two digits after the decimal point.

The two formulas say the same thing as the code. The variable total_size_bytes is an intermediate value, and total_size_mb is the final value that the printed line depends on. Because each step feeds the next, an error introduced early in the chain carries forward into every later value. A wrong parameter count produces a wrong byte count, and a wrong byte count produces a wrong megabyte figure, even when the conversion step itself is written correctly.

Before you read the calculation further, make a prediction. Suppose a model has 10,000,000 parameters. Using only the two steps above (multiply by 4, then divide by 1,048,576) and the two-decimal print format, decide what number the last line will print.

The word parameter refers to the stored numbers a module learns. A module's parameters() method hands back a sequence of tensors, and each tensor is a block of numbers. The numel() call measures how many individual numbers sit inside one tensor. Summing those counts over every tensor in the module gives the number of parameters that module contributes to the model.

Read the expression from the inside out. The part p.numel() asks one tensor how many values it holds. The generator expression applies that question to every tensor in the module. The outer sum adds the answers. The attention line has the same shape, and the only difference is which module it asks about. Because the two lines are identical except for block.ff and block.att, any disagreement between the two results must come from the module being measured, not from the counting method.

This is why the count is a sum and not a single lookup. A module is usually made of several layers, and each layer contributes its own tensors. The loop inside the generator expression visits each of those tensors in turn, and the running total is complete only after the last one has been added. The total grows once per tensor, so when a count seems too small, check whether every tensor in the module was visited. Biases are part of the count as well, so a layer's total includes them along with its weights.

Before running the counting code, decide which of the two modules you expect to be larger. Think about what each module does in a transformer block and what kind of weights it would need to hold. The source gives a useful benchmark: the feed forward module contains approximately twice as many parameters as the attention module. If your computed counts reverse that relationship, treat it as a sign to check the code and the configuration rather than accepting the result.

When the printed size looks wrong, do not start by rechecking the final line. Inspect the intermediate values in order: total_params, then total_size_bytes, then total_size_mb. The relationships they must satisfy are simple. total_size_bytes should be exactly four times total_params, and total_size_mb should be total_size_bytes divided by 1,048,576. The first value that breaks one of these relationships marks the step where the error was introduced.

The same check applies when a module count disagrees with expectation. Look at the module being measured, the counting expression, and the configuration that built the module. Because the counting expression is identical for every module, a changed result usually points to the module selected or the configuration used to build it, not to the arithmetic.

Everything above used a single configuration, GPT_CONFIG_124M. That configuration is a dictionary that describes one GPT model. The same model class can produce other sizes if you give it a different dictionary, so you do not need a new class for each size. The work is in building the right dictionary. Larger GPT models are built by copying the base configuration dictionary and overriding only the size fields, such as the embedding dimension, the number of layers, and the number of heads.

Once the full model is built from the edited configuration, the same reporting approach is applied to the whole GPTModel. The output reports three figures: a total parameter count, a count of trainable parameters once weight tying is taken into account, and the total size of the model in megabytes. Weight tying can make the trainable count lower than the total, so the two figures are not interchangeable. The size is still derived from the total parameter count using the same 4-byte and 1,048,576 steps described earlier.