Foundation Models and Finetuning

Foundation models are very large, and that size is where the trouble starts. Because of their scale, memory is a bottleneck for working with them. This is true in two situations: when you run a model to produce outputs, which is called inference, and when you adjust a model for a specific task, which is called finetuning.

Keep both situations in mind as you read. The goal of this article is to trace where memory goes, why finetuning asks for much more of it than inference, and which two techniques, parameter-efficient finetuning and quantization, are designed to reduce it.

Inference uses a model to produce outputs. Finetuning uses a model as a starting point and adjusts it for a specific task. The source is clear that finetuning typically needs much more memory than inference for the same model. So when you move from running a model to finetuning it, the memory question changes from whether the model fits to whether the extra work of adjusting it also fits.

This difference is the core of the memory bottleneck. The model itself is the same in both cases. What differs is the additional state that finetuning needs, and the next section names the three factors that drive the total.

During finetuning, the source identifies three key contributors to a model's memory footprint. The first is the number of parameters, which describes the model itself. The second is the number of trainable parameters, which describes how many of those parameters are adjusted during finetuning. The third is the numerical representations, which describe how many bits are used to store each value.

The three contributors are not interchangeable, and that is what makes them useful to reason about. The number of parameters is a property of the model. The number of trainable parameters is a property of the finetuning process, because it counts how many parameters are adjusted. The numerical representations are a property of storage, because they set how many bits each value uses. Each one feeds into the total, so reducing any one of them can reduce the footprint.

Parameter-efficient finetuning, usually called PEFT, reduces the memory needed for finetuning by reducing the number of trainable parameters. It leaves the model's overall parameter count where it is. What it changes is how many of those parameters are adjusted when the model is finetuned.

Read the diagram as a split. Most of the pretrained model stays fixed, and only a reduced set of parameters is adjusted. Because the memory cost of finetuning grows with the number of trainable parameters, shrinking that set shrinks the cost. The model still has the same number of parameters, so PEFT is aimed at the second contributor, not the first.

Quantization targets the third contributor, the numerical representations. It reduces memory by converting a model to fewer bits per value. The source gives 16 bits, 8 bits, and even 4 bits as examples of the lower widths used. Fewer bits per value means each stored number occupies less space, so the whole model takes up less memory without changing how many parameters it has.

Quantization fits inference most naturally. The source explains that inference is typically done using as few bits as possible. Training is different. It is more sensitive to numerical precision, which makes it harder to train a model in lower precision. The table below sets the two side by side.

Training is typically done in mixed precision. Some operations run in higher precision, such as 32-bit, and others run in lower precision, such as 16-bit or 8-bit. The mix lets training use fewer bits in places while keeping higher precision where the source says it is needed for stability of training.