Model Size Calculation

A model's parameters are the numbers that make up the model, and a program can estimate how much memory those numbers occupy. The estimate is a short chain of arithmetic: start from a total parameter count, turn it into a number of bytes, and then turn the bytes into megabytes. Each step answers a different question, and the final printed value depends on all of them. This article follows that chain one step at a time, so that you can predict the printed result and locate the step responsible if the result looks wrong.

The program has three calculation steps, and each one answers a different question. The first step asks how many bytes the parameters need. The program assumes every parameter is a float32 number, which takes 4 bytes, so the byte count is the parameter count multiplied by 4. The second step converts bytes into megabytes by dividing by 1024 * 1024. The third step is the print statement, which formats the megabyte value with two digits after the decimal point.

The code above follows the three steps described in the source, with the same 10,000,000 parameter count used in the prediction. Because the value of total_size_mb depends on total_size_bytes, which in turn depends on total_params, the three variables form one chain. An error introduced at any point in the chain is carried into every later value.

The first step rests on one assumption: every parameter is a float32 number, and a float32 value takes 4 bytes. That is why the byte count is always the parameter count multiplied by 4. The assumption is fixed in the program rather than read from the model, so the program's result is an estimate built on that assumption. If the parameters were stored in a different format, the multiplier would change, and the same chain would produce a different size.

The second step divides the byte count by 1024 * 1024, which is 1,048,576. The divisor is not one million. Using one million gives a different and incorrect result, because one megabyte in this program is defined as 1,048,576 bytes. Written as a calculation, the conversion for the 10,000,000 parameter model is $\frac{40{,}000{,}000}{1{,}048{,}576} \approx 38.1470$, which the print line rounds to 38.15.

The source names two modules as a key comparison: the feed forward module and the multi-head attention module. To compare them fairly, count both with the same method. Count the parameters of each layer, including its biases. Add the layer counts to get a module total. Only after every module total is computed should the totals be compared with each other. The source does not give specific counts for these two modules, so the method, not a memorized number, is what carries forward.

The same method applies at both levels. For a single layer, the parameter count includes its weight terms and its bias terms. For a module, the module total is the sum of the counts of the layers that belong to it. Because both levels use the same rule, a module total can be compared with another module total directly. A comparison that mixes methods, such as counting weights for one module and weights plus biases for the other, is not a fair comparison.

When the printed size looks wrong, do not start by rechecking the final line. Inspect the intermediate values in order: total_params, then total_size_bytes, then total_size_mb. Two relationships must hold. The first is that total_size_bytes should be exactly four times total_params. The second is that total_size_mb should equal total_size_bytes divided by 1,048,576. The first value that breaks one of these relationships marks the step where the error was introduced. Later values can look plausible, but they inherit the error from that point on.