Parameter-Efficient Finetuning

A natural first guess about quantization is that storing a model with fewer bits must make it faster. Smaller values mean less data to move and a smaller memory footprint, so this idea has a real basis. But it is not a guarantee: converting values between formats can add work, and changing their precision can change the values themselves. These same trade-offs matter when deciding how to finetune a model. This article follows both questions: what happens when model values use fewer bits, and what can be trained when updating every parameter is too costly?

Inference quantization reduces the precision of a model's parameters and values while the model is being served. A value stored in a wider format can be converted to one of the fewer values representable in a narrower format. The converted value may differ slightly from the original. If a value lies outside the narrower format's range, it can become infinity or an arbitrary value. Quantization can therefore reduce the model's memory footprint, but quality must be checked after conversion.

The diagram shows the representational change, not a particular numeric conversion rule. The source describes formats such as FP8 and FP4, which are floating-point formats, and INT8 and INT4, which are integer formats. The narrower representation uses fewer bits, but fewer bits also mean fewer possible values. That is why smaller storage and potentially faster computation come with a quality trade-off.

Reduced precision can help in two linked ways. A smaller model footprint can leave room for a larger batch, allowing more inputs to be processed in parallel. Reduced precision can also speed up computation itself. Either effect can contribute to lower inference latency, but the outcome depends on the work required by the actual serving path.

The important question is not just how many bits the stored weights use. It is also whether the serving path can use that representation efficiently. If values must be converted to another format, that conversion adds time and can offset some of the speed benefit. So fewer bits can mean less data to move and faster computation, but they do not guarantee lower latency in every setup.

Training-related uses of reduced precision have different goals. Quantization-aware training aims to help a model retain quality when it later runs at low precision. Direct lower-precision training aims to perform training computations at lower precision, which can improve training efficiency but is harder because backpropagation is more sensitive to precision. Mixed precision assigns different precisions to different values or parts of training. These approaches should not be treated as interchangeable.

In quantization-aware training, the model simulates low-precision behavior while the computation is carried out in high precision. This can prepare the model for better low-precision inference, but it does not itself reduce training time. Direct lower-precision training is different: the training process uses lower precision, which can help with training efficiency, but the lower-precision learning signal can be disturbed during backpropagation.

Mixed precision does not require every value to use the same precision. For example, a higher-precision copy of weights can be kept while gradients and activations use lower precision. Precision can also vary according to sensitivity, with more-sensitive values kept at higher precision. The source gives LLM-QAT as an example: weights and activations are quantized to 4 bits while embeddings remain at 16 bits.

One bit is the lower limit for representing a value with bits, and research has explored 1-bit representations. In 2024, Microsoft researchers introduced BitNet b1.58, a transformer-based language model described as requiring 1.58 bits per parameter. The reported comparison found its performance comparable to 16-bit Llama 2 up to 3.9B parameters.

That result has boundaries. It is a comparison of benchmark performance, not a claim that BitNet b1.58 is faster. The reported gap also varies from benchmark to benchmark. The useful lesson is that very low-bit models are possible, not that the same performance or speed is guaranteed for every model size or task.

Full finetuning updates every parameter in the model. Its memory cost is more than the model weights alone: a training run also needs memory for gradients, optimizer states, and activations. The source identifies these components but does not provide a separate numeric memory estimate for each one, so a weights-only calculation should not be mistaken for the total training requirement.

For the same 7B model in a 16-bit format, the source estimates about 56 GB for full finetuning before activations. That total is substantially more than the weights-only estimate because training includes other memory costs, such as gradients and optimizer states. Use the 14 GB calculation as a way to estimate just the stored weights; do not use it as a GPU-memory budget for a full finetuning run.

Partial finetuning changes the number of parameters that are updated: some are trainable, while the rest are frozen and keep their values throughout training. This reduces the trainable parameter count, and therefore reduces memory use. The training-related memory burden associated with updating parameters is consequently smaller, although the source does not give a separate gradient or optimizer-state estimate for each frozen or trainable group.

Freezing is not automatically parameter-efficient. It can reduce the number of trainable parameters while still requiring many of them to approach full-finetuning performance. The source reports a BERT large study in which roughly 25% of the parameters had to be updated to match full finetuning on the GLUE benchmark.

Parameter-efficient finetuning, or PEFT, aims to get close to full-finetuning performance while using several orders of magnitude fewer trainable parameters. There is no official cutoff that defines the label. The emphasis is on achieving strong performance with far fewer parameters being updated.

Adapters are one PEFT approach. Instead of updating the original model weights, adapter finetuning keeps those weights frozen and trains small modules inserted into the model. In the approach described by Houlsby and colleagues, two adapter modules were placed inside each transformer block of a BERT model. During finetuning, only the adapter parameters change.

The diagram traces the adapter's role: the input goes through the frozen base computation and the adapter path, and their outputs are combined before continuing. The adapter adds trainable parameters without requiring the original weights to be updated. In the reported GLUE result, the adapter arrangement came within 0.4% of full finetuning while using only 3% of the trainable parameters.

PEFT can also lower the hardware and data burden of finetuning. The source says full finetuning may require tens of thousands to millions of examples before quality improves noticeably, while some PEFT methods can produce strong results with only a few thousand examples. That is a general claim about some PEFT methods, not a guarantee that every adapter setup will need that amount of data.