Effect of Trainable Parameters on Memory
Weight memory equals the parameter count multiplied by the bytes per parameter; a 13B model at 2 bytes per parameter needs 26 GB for its weights.
Why Memory Estimates Come First
Before you run a large model, you need an estimate of how much memory it will occupy. Large language models have billions of parameters, and each parameter has to be held in memory while the model works. A good estimate tells you whether a model will run at all, and whether you need more hardware or a smaller model. The encouraging part is that the estimate is simple arithmetic built from two numbers: how many parameters the model has, and how many bytes each parameter takes to store.
What do you think happens?
A 13B-parameter model stores its weights at 2 bytes per parameter and needs 26 GB. How much memory do the weights of a 70B-parameter model need at the same 2 bytes per parameter?
Reveal answer
140 GB
Multiply the parameter count by the bytes per parameter: 70B × 2 bytes = 140 GB. Because the calculation is linear, the answer scales directly with the parameter count.
Weight Memory From Parameter Count
A model's weights are stored as a long list of numbers, with one entry per parameter. The storage cost of the weights is therefore the number of parameters multiplied by the number of bytes used to store each one. The result is usually expressed in gigabytes, and the parameter count is written with a B for billions.
In this formula, N is the number of parameters and b is the number of bytes each parameter occupies. The choice of b matters a great deal. The same model can have very different footprints depending on how its values are stored, because halving b halves the memory needed for the weights.
Weight memory for a 13B model
A model has 13B parameters, and each parameter is stored in 2 bytes. How much memory do the weights need?
- 1Write the formula
Use $\text{Weight memory} = N \times b$, with N as the parameter count and b as the bytes per parameter.
- 2Substitute the values
$N = 13\text{B}$ and $b = 2$ bytes, so the calculation is $13\text{B} \times 2 \text{ bytes}$.
- 3Compute the product
$13 \times 2 = 26$, so the weights need 26 GB.
The weights of a 13B model at 2 bytes per parameter need 26 GB.
How Weight Memory Scales
The formula is linear in the parameter count, so doubling the model roughly doubles its weight memory. This growth is what makes memory a bottleneck for operating large models. The 70B example makes the point clearly. Its weights alone need $70\text{B} \times 2 \text{ bytes} = 140$ GB, which is far more than the 26 GB needed by the 13B model. The overhead factor has not even been applied yet at this size.
| Model | Parameters | Bytes per parameter | Weight memory |
|---|---|---|---|
| 13B model | 13B | 2 | 26 GB |
| 70B model | 70B | 2 | 140 GB |
| 13B model (hypothetical 1-byte storage) | 13B | 1 | 13 GB |
Weight memory scales with both the parameter count and the bytes per parameter. The third row is a generated illustration showing that halving b halves the result.
Overhead Factor and Inference Total
Weight memory is not the whole story for inference. The total inference memory is the weight memory multiplied by an overhead factor of 1.2. For the 13B example, the weights need 26 GB, so the total inference memory is $26 \times 1.2 = 31.2$ GB. The diagram below follows the same two steps: the weights form the starting point, the overhead factor is applied to them, and the result is the total inference estimate.
The overhead factor adds a margin of 20 percent on top of the weights. Its job in the estimate is to cover memory beyond the raw parameter storage, and the factor is applied as a multiplier to the weight memory rather than as a separate calculation.
Values Stored During Training
Inference needs only the weights and their overhead. Training is different. When you update the parameters, the system must also keep information about each trainable parameter. The source uses the Adam optimizer as its example, and each trainable parameter has three values stored for its gradient and optimizer states. If each of those values takes 2 bytes, each trainable parameter needs $3 \times 2 = 6$ bytes of additional storage on top of its weight.
The extra storage for gradients and optimizer states is described by a separate term that sits alongside the weight memory. It counts only the training-specific values, so the two terms are added together rather than merged.
- 1N_{trainable} is the number of trainable parameters, the only part of the model whose gradients and optimizer states are stored.
- 2k is the number of values stored per trainable parameter for gradients and optimizer states; in the Adam example, k = 3.
- 3b is the number of bytes per stored value; in the example, b = 2.
- 4The product N_{trainable} × k × b gives the gradient and optimizer memory, which is separate from the weight memory calculated earlier.
Training memory for a fully trainable 13B model
All 13B parameters are trainable with the Adam example (k = 3) and 2 bytes per value. How much memory do the gradient and optimizer states need?
- 1Identify the inputs
$N_{\text{trainable}} = 13\text{B}$, $k = 3$, and $b = 2$ bytes.
- 2Multiply the terms
$13\text{B} \times 3 \times 2 = 78$ GB.
- 3Add the weight memory
The weights still need their 26 GB from before. Adding the two separate terms gives $26 + 78 = 104$ GB as a generated total for weights plus gradient and optimizer storage.
The gradient and optimizer states need 78 GB, which is three times the 26 GB of weights in this example.
Reducing Trainable Parameters
The number of trainable parameters is the lever that controls gradient and optimizer memory. The weight memory is not affected by this lever. The before-and-after view below shows the 13B case when all parameters train compared with a case where only 1B parameters train. The weights keep their 26 GB in both cases, while the gradient and optimizer storage shrinks.
Gradient and optimizer memory with 1B trainable parameters
Only 1B of the model's parameters are trainable, with k = 3 and 2 bytes per value. How much gradient and optimizer memory is needed, and what is the change from the full 13B case?
- 1Apply the same rule
$1\text{B} \times 3 \times 2 = 6$ GB for gradients and optimizer states.
- 2Compare with the full model
The full 13B case needed 78 GB, so the reduction is $78 - 6 = 72$ GB.
- 3Check the weights
The weights still need 26 GB, so the training total in this case is $26 + 6 = 32$ GB, compared with the generated total of 104 GB when all 13B parameters train.
Reducing trainable parameters from 13B to 1B cuts gradient and optimizer memory from 78 GB to 6 GB, but leaves the 26 GB of weights unchanged.
Mistakes in Memory Estimates
Common mistake Reporting the weight memory as the total inference memory
What it looks likeStopping at 26 GB for the 13B model.
Why it failsThe overhead factor of 1.2 is applied to the weights before the inference total is known.
Do this insteadMultiply the weight memory by 1.2 to get 31.2 GB for the 13B example.
Common mistake Counting only one value per trainable parameter during training
What it looks likeComputing training memory as N × b, which gives 26 GB for 13B at 2 bytes.
Why it failsThe Adam example stores three gradient and optimizer values per trainable parameter, which is the k term in the formula.
Do this insteadUse $N_{\text{trainable}} \times k \times b$ with k = 3 to get 78 GB for the gradient and optimizer states.
Common mistake Believing that freezing parameters removes the weight memory
What it looks likeAssuming that training only 1B parameters brings the total down to 6 GB.
Why it failsThe model still has all of its parameters, so the weights still need their full storage.
Do this insteadAdd the 26 GB of weights to the 6 GB of gradient and optimizer storage for a total of 32 GB in that case.
Common mistake Forgetting that bytes per parameter changes the result
What it looks likeUsing 2 bytes per parameter for every model, whatever the storage format.
Why it failsHalving b halves the weight memory, so the same parameter count can have very different footprints.
Do this insteadCheck the bytes-per-parameter value for the storage format before calculating.
Estimating a 7B Model
A 7B-parameter model stores each parameter in 2 bytes. First, calculate its weight memory and its total inference memory with the overhead factor of 1.2. Then suppose all 7B parameters are trainable with the Adam example (k = 3) at 2 bytes per value, and calculate the gradient and optimizer memory. Explain which of your totals would change if only 1B parameters were trainable.
Hints
- Start with the weight memory formula: parameters multiplied by bytes per parameter.
- The inference total multiplies the weight memory by 1.2.
- For training, multiply the trainable parameter count by k and by b, and keep that result separate from the weights.
- Only the gradient and optimizer term depends on the trainable count; the weight memory stays the same.
Recap of Memory Effects
- 1Weight memory equals the parameter count multiplied by the bytes per parameter, so the 13B example at 2 bytes needs 26 GB.
- 2Total inference memory is the weight memory multiplied by the overhead factor of 1.2, giving 31.2 GB for the 13B example.
- 3Training adds gradient and optimizer storage of N_trainable × k × b; with k = 3 and 2 bytes per value, the 13B model needs 78 GB.
- 4Reducing trainable parameters from 13B to 1B cuts gradient and optimizer memory to 6 GB, but the weights still occupy their full 26 GB.
Key Takeaways
- Weight memory equals the parameter count multiplied by the bytes per parameter; a 13B model at 2 bytes per parameter needs 26 GB.
- Total inference memory is the weight memory multiplied by the overhead factor of 1.2, giving 31.2 GB for the 13B example.
- Training stores k values per trainable parameter for gradients and optimizer states; with the Adam example (k = 3) at 2 bytes each, 13B trainable parameters need 78 GB.
- Reducing trainable parameters lowers gradient and optimizer memory, but the weights keep their full storage.