Parameter-Efficient Finetuning (PEFT)
Memory is a bottleneck for foundation models during both inference and finetuning, and finetuning typically needs much more memory than inference.
Why Scale Strains Memory
Foundation models are very large, and that size makes memory a bottleneck for working with them. The bottleneck appears in two situations: when you run a model to produce outputs, which is called inference, and when you adjust a model for a specific task, which is called finetuning. Finetuning typically needs much more memory than inference, so a model that runs comfortably may still be impossible to finetune on the same hardware.
Keep this picture in mind as you read. The question this article answers is not why models are large, but how the memory a model occupies is determined and which levers reduce it. Two techniques pull those levers: parameter-efficient finetuning (PEFT) and quantization.
Three Contributors to the Memory Footprint
A model's memory footprint is the amount of memory it occupies. During finetuning, three key contributors feed into that total: the number of parameters, the number of trainable parameters, and the numerical representations used to store the model.
These three contributors are not interchangeable, and each answers a different question. The number of parameters describes the model itself. The number of trainable parameters describes how many of those parameters are adjusted during finetuning. The numerical representations describe how many bits are used to store each value. Because each one feeds into the total, reducing any one of them can reduce the footprint.
The reasoning behind the three contributors is direct. The more trainable parameters a model has, the higher its memory footprint. So if you can reduce the number of parameters that are updated during finetuning, you reduce the memory needed for finetuning. That observation is the motivation for PEFT.
Training Fewer Parameters with PEFT
PEFT reduces the memory needed for finetuning by reducing the number of trainable parameters. The model keeps its full set of parameters, but only a smaller set is updated during finetuning. The parameters that are not updated stay as they are, and the trainable parameters are the only ones whose values change as the model adapts to the task.
Notice what changes in the diagram. The total model is unchanged, but the portion that must be adjusted during finetuning is smaller. Since memory use during finetuning grows with the number of trainable parameters, shrinking that portion is how PEFT lowers the memory needed to adapt the model.
Quantization and Bits per Value
Quantization reduces memory by converting a model to fewer bits per value. The bit width is the third contributor from the list above: it describes how many bits store each numerical value in the model. Storing the same number of values in fewer bits means the model occupies less memory overall.
Converting a 13 billion parameter model
The source gives a concrete case: a 13 billion parameter model takes 52 GB of memory in one numerical format and 26 GB after quantization. What does the change show?
- 1Identify the starting size
The model occupies 52 GB before quantization.
- 2Identify the ending size
After converting the model to fewer bits per value, it occupies 26 GB.
- 3Compare the two sizes
The memory is halved, from 52 GB down to 26 GB, while the number of parameters stays the same.
- 4Connect to the contributors
The parameter count did not change, so the reduction comes from the numerical representation, the third contributor.
Reducing the bits used per value cut the model's memory from 52 GB to 26 GB without removing any parameters.
What do you think happens?
A 13 billion parameter model takes 52 GB at its original precision. If quantization halves the bits used per value and the parameter count stays the same, how much memory does the model occupy afterward?
Reveal answer
26 GB
The source reports that quantization moves this model from 52 GB to 26 GB. Halving the bits per value halves the memory the weights occupy, because the number of values is unchanged.
Inference Precision Versus Training Precision
Quantization fits inference more naturally than training. Inference is typically done using as few bits as possible, and the source gives 16 bits, 8 bits, and even 4 bits as examples. Training is more sensitive to numerical precision, which makes it harder to train a model in lower precision.
| Aspect | Inference | Training (finetuning) |
|---|---|---|
| Typical precision | As few bits as possible, for example 16, 8, or 4 bits | Mixed precision: some operations at 32-bit, others at 16-bit or 8-bit |
| Sensitivity to precision | Less sensitive to reduced precision | More sensitive; harder to train in lower precision |
| Motivation for lower bits | Reducing memory when producing outputs | Using fewer bits where stability allows it |
Precision used for inference compared with precision used for training
Training is typically done in mixed precision. Some operations run at higher precision, such as 32-bit, while others run at lower precision, such as 16-bit or 8-bit. This mix lets training use fewer bits in places while keeping higher precision where it is needed for training stability. The result is that the bit width is not a single fixed choice during training; it varies by operation.
Mistakes Learners Make About Memory
Common mistake Assuming finetuning needs about the same memory as inference
Why it failsThe source states that finetuning typically needs much more memory than inference for the same foundation model.
Do this insteadTreat finetuning as the heavier workload and plan memory for the additional demands it places on the system.
Common mistake Thinking PEFT removes parameters from the model
What it looks likeA model with a fixed parameter count can still be finetuned with PEFT.
Why it failsPEFT reduces the number of trainable parameters, not the number of parameters the model has.
Do this insteadKeep the two counts separate: total parameters describe the model, and trainable parameters describe what is updated during finetuning.
Common mistake Assuming quantization should be applied the same way in training and inference
Why it failsInference tolerates fewer bits well, while training is more sensitive to numerical precision and often uses mixed precision.
Do this insteadApply aggressive bit reduction with inference in mind, and expect training to keep higher precision for some operations.
Common mistake Treating the three contributors as a single number rather than three separate factors
Practice Estimating Memory Changes
A team is finetuning a foundation model. They can either apply PEFT so that fewer parameters are trainable, or quantize the model so each value uses fewer bits. For each option, name which of the three contributors it targets, and explain which option would be the better choice if the team's main goal is to reduce memory during finetuning. Justify your answer using the source's reasoning.
Hints
- Map PEFT to one of the three contributors and quantization to another.
- Recall that the source names the number of trainable parameters as the direct motivation for PEFT.
- Consider that quantization is most natural for inference, while training is more sensitive to precision.
Key Takeaways
- Memory is a bottleneck for foundation models during both inference and finetuning, and finetuning typically needs much more memory than inference.
- Three contributors shape the memory footprint during finetuning: the number of parameters, the number of trainable parameters, and the numerical representations used to store the model.
- PEFT reduces finetuning memory by reducing the number of trainable parameters, and quantization reduces memory by converting a model to fewer bits per value, as shown by the move from 52 GB to 26 GB for a 13 billion parameter model.
- Inference is typically done with as few bits as possible, while training is more sensitive to precision and typically uses mixed precision with some operations at 32-bit and others at 16-bit or 8-bit.