Model Memory Footprint During Inference

Before you run a large model, you need an estimate of how much memory it will occupy. Large language models have billions of parameters, and each parameter has to be held in memory while the model works. A good estimate tells you whether a model will run at all, and whether you need more hardware or a smaller model.

The good news is that the estimate is simple arithmetic built from two numbers: how many parameters the model has, and how many bytes each parameter takes to store. This article builds that arithmetic step by step, then extends it to training.

A model's weights are stored as a long list of numbers, one entry per parameter. The storage cost of the weights is therefore the number of parameters multiplied by the number of bytes used to store each one. The result is usually expressed in gigabytes, and the parameter count is written with a B for billions.

In this formula, N is the number of parameters and b is the number of bytes each parameter occupies. The choice of b matters a great deal. Halving b halves the memory needed for the weights, so the same model can have very different footprints depending on how its values are stored.

The 4-byte row above is a generated variation on the source's 2-byte example, calculated with the same formula: 13 × 4 = 52 GB.

Weight memory is not the whole story for inference. The source applies an overhead factor of 1.2 on top of the weights. Total inference memory is the weight memory multiplied by that factor.

Inference only needs the weights and their overhead. Training is different. When you update the parameters, the system must also keep information about each trainable parameter. The source uses the Adam optimizer as its example, and each trainable parameter has three values stored for its gradient and optimizer states.

Each trainable parameter therefore needs 3 × 2 = 6 bytes of additional storage on top of the weights. The general form uses k for the number of values stored per parameter and b for the bytes per value.

In this formula, N_{trainable} is the number of trainable parameters, k is the number of values stored per parameter for gradients and optimizer states (three in the Adam example), and b is the number of bytes per value. This term covers only gradient and optimizer storage. It is separate from the weight memory calculated earlier.

The number of trainable parameters is the lever that controls gradient and optimizer memory. The per-value rule stays the same; only N_{trainable} changes. If only 1B of the parameters are trainable, the same rule gives 1B × 3 × 2 bytes = 6 GB for gradients and optimizer states. That is a large reduction from the 78 GB needed when the full 13B model trains.

Notice that the weight memory does not disappear when fewer parameters train. The model still has all of its parameters, so the weights still need their full 26 GB of storage. Freezing parameters only removes the gradient and optimizer storage tied to the frozen values.