Model-level techniques for inference optimization

Training a model is only part of the story. When a trained model answers requests in production, its speed and cost depend on two things: the model itself, and the way the serving system is organized around it. The same model can feel fast or slow, and can be cheap or expensive to run, depending on how the system is built. This article is about the techniques that make inference faster and cheaper, and about how those techniques are sorted into families.

The article focuses on one part of that toolkit: model-level techniques, and the parallelism strategies that sit on the other side of the split. You will see how replica parallelism and tensor parallelism differ, why neither one is always the right answer, and how a single word, memory-bound, can mean different things depending on who is using it.

Inference optimization techniques fall into two broad families. The first family works on the model. Quantization and distillation are model-agnostic, meaning they can be applied across different model architectures. Other techniques are tailored to architecture. Because the attention mechanism is a key bottleneck in transformer models, many optimizations target attention directly, including KV cache management and writing attention kernels.

The second family works at the level of the inference service: how requests are grouped together and how work is spread across hardware. Batching and parallelism strategies belong here. Parallelism strategies are the subject of the rest of this article.

The source is explicit that no single technique is always the right one. The choice depends on the workload and on the performance requirements. Among the techniques it singles out as most impactful across use cases are quantization, tensor parallelism, replica parallelism, and attention mechanism optimization. Two of those four, tensor parallelism and replica parallelism, are the parallelism strategies examined here.

Replica parallelism is the simplest strategy to picture. Before reading on, try to reason it out. Suppose a model is served from one machine, and you decide to run more copies of it on additional machines to lower response times. Think about what happens to the number of requests each machine has to handle.

Replica parallelism runs several full copies of the model. Each copy serves a share of the incoming requests. Because each machine handles fewer requests, each request gets more resources, and response time improves. The benefit is paid for in hardware: scaling up replica parallelism costs more machines. The source ties this trade-off directly to priorities. If low latency matters more than cost, scaling up replica parallelism is the natural move.

The source also describes replica parallelism as relatively straightforward to implement, which is one reason it appears among the most impactful techniques it lists.

Tensor parallelism is the second strategy the source ranks among the most impactful. It is credited with two named benefits: it reduces latency, and it enables serving larger models. The second benefit is the key contrast with replica parallelism. Replica parallelism adds machines so that each handles fewer requests. Tensor parallelism is framed around the model itself and the latency of a single response, not around spreading whole requests across identical copies.

The source does not describe how tensor parallelism divides the work internally, so this article does not explain that mechanism. Take away the pair of effects it is credited with. A model too large for a single machine can be served at all, and the response can arrive faster. Both points are worth remembering when comparing it with replica parallelism.

Before choosing a strategy, it helps to clear up a term that appears constantly in inference discussions: memory-bound. A common assumption is that the phrase has one fixed meaning. The source reports that this assumption does not hold. The word is used with different meanings depending on the speaker's background.

Inference costs are tied to the economics of the model itself. The source explains that inference involves the forward pass, while training involves both the forward and backward passes. It then describes a relationship between training cost and inference cost from the viewpoint of a model provider. Trace the variables as they are used in that relationship.

Here T stands for the training cost, p for the value recovered from each inference call, and N for the number of inference calls. The source reads this inequality as a condition for developing a model to make sense at all: the money recovered from inference must exceed the training cost.