Post-Training Quantization

A common first reaction to quantization is that storing a model with fewer bits must make it faster. Fewer bits means less data to move, less memory to fill, and, it seems, less arithmetic to perform. That reasoning is partly right, which is exactly why it is easy to over-trust. This article starts from that belief, tests it, and then builds up the more accurate picture that the rest of the topic depends on.

In the early days of deep learning, models were trained and served using 32-bit floating-point values, written FP32. Since the late 2010s it has become increasingly common to serve models at 16 bits and below. The motivation is that quantization can improve computation speed and reduce the memory footprint of the model. The memory footprint is the amount of storage the model's values occupy while it runs.

Once you reach 8 bits and below, the choice of number format becomes more delicate. One option is to keep values as floats in a minifloat format, such as FP8 with 8 bits or FP4 with 4 bits. The more common option is to convert parameter values into an integer format, such as INT8 or INT4. The table below lists the formats mentioned in the source material, ordered from widest to narrowest.

Reducing the width of stored values changes what each value can represent. The diagram below shows the change in stored parameter values. Before quantization, a parameter occupies a wide floating-point slot. After quantization, the same parameter is held in a narrower integer slot. The values are not identical after conversion, which is the central trade-off of this topic.

The diagram is an illustration of the general idea rather than a specific format. A value between two levels is assigned to one of them, so the stored result is an approximation. Each conversion can shift values slightly. Values that fall outside a format's range are a more serious problem, because they can become infinity or an arbitrary value. For that reason, quality has to be checked after quantization rather than assumed.

Reduced precision helps in two linked ways. First, a smaller footprint allows a larger batch size, so the model can process more inputs in parallel. Second, reduced precision speeds up the computation itself, which lowers inference latency. Latency is the time it takes to produce a response for one request. The arithmetic argument is easiest to see with a bit-by-bit model of addition, which is why the source uses it as the starting point for the mechanism.

Reduced precision does not always reduce latency. The reason is format conversion overhead in the inference path. The pipeline below follows a stored low-precision weight from memory to the compute unit. Each step that changes the format is a place where time can return, even though the stored values are smaller.

The two conversion steps in the pipeline are where the gain can be lost. Moving fewer bits is a real saving, but a conversion before the arithmetic and a conversion after it can consume that saving. This is why the simple rule that fewer bits means faster does not hold in every case. The benefit has to be measured on the actual serving path, not inferred from the bit width alone.

Post-training quantization is only one of several ways lower precision enters the model lifecycle. The table below sets the four approaches side by side. The key question for each is at what stage reduced precision is applied, and whether it is aimed at inference quality, training cost, or both.

Quantization during training is less common than PTQ, but it is gaining traction. It has two distinct goals, and a given technique may serve one, both, or neither. The subsections below take each approach in turn.

Quantization has a limit. A value cannot use fewer than 1 bit, and several efforts have tried 1-bit representations, including BinaryConnect, Xnor-Net, and BitNet. In 2024, Microsoft researchers declared the era of 1-bit LLMs by introducing BitNet b1.58, a transformer-based language model that requires only 1.58 bits per parameter.

The number 1.58 is not a bit pattern that a single stored bit can hold. It describes information content: a weight that can take one of three values, such as -1, 0, or +1, carries $\log_2 3 \approx 1.58$ bits of information. The diagram below shows the three states. A few states, arranged across a whole weight matrix, are what allow the representation to approximate full-precision weights.

The reported performance of BitNet b1.58 is comparable to 16-bit Llama 2 up to 3.9B parameters. A benchmark comparison with Llama models at several sizes is the source's evidence for this claim. Read it with care: the comparison is about benchmark scores, not about speed, and the gap varies from benchmark to benchmark. A benchmark result does not, by itself, show that the model runs faster.