Inference service-level techniques: batching and parallelism

Training a model is only part of its life. Once a trained model is put to work answering requests, its speed and its cost depend on two things: the model itself, and the way the serving system is organized around it. Two teams can run the very same model and get very different response times and bills, simply because they arranged their machines differently.

Inference optimization is usually organized into two broad families. The first family works on the model. Quantization and distillation are model-agnostic techniques, which means they can be applied across different model architectures. Other model-level techniques are tailored to the architecture. Because the attention mechanism is a key bottleneck in transformer models, many of these optimizations target attention directly, including KV cache management and writing attention kernels.

The second family works at the level of the inference service: how requests are grouped together and how work is spread across hardware. This family includes batching and parallelism strategies. The parallelism strategies are the subject of this article.

Replica parallelism is the simplest strategy to picture. Before reading on, try to reason it through yourself. Suppose one machine serves a model and receives a steady stream of requests. Each request has to wait for that machine's resources. Now suppose you run more replicas of the model on additional machines and share the same load among them. Follow what happens to each machine as the replica count grows.

Replica parallelism adds machines so that each one handles fewer requests. Its trade-off is explicit in the source: scaling it up costs more machines, but it buys lower latency. The source ties the choice to priorities. If low latency matters more than cost, scaling up replica parallelism is the natural move. The source also describes replica parallelism as relatively straightforward to implement, which is one reason it appears among the most impactful techniques.

Tensor parallelism is credited with two named benefits: it reduces latency, and it enables serving larger models. The second benefit is the one replica parallelism cannot offer in the same way. A model too large for a single machine can be served at all. Notice how the framing differs. Replica parallelism is described in terms of spreading requests across identical copies. The source's description of tensor parallelism is framed around the model itself and the latency of a response.

The general principle behind the source's advice is that the best technique depends on what you are trying to optimize. Imagine two teams serving the same model. A team whose main goal is low latency would lean toward adding replicas. A team whose main goal is keeping costs down would reach a different conclusion, because adding machines is exactly the step that raises cost. Neither team is wrong. They are optimizing for different things.

Before choosing between strategies, one term needs clearing up because it appears constantly in inference discussions: memory-bound. It is easy to assume the phrase has one fixed meaning, so that any engineer who calls a workload memory-bound means the same thing as any other. The source reports that this assumption does not hold. The word is used with different meanings depending on the speaker's background.

So when you read or hear the phrase, the question to ask is which resource is the limit. Is the constraint how fast data can move, which is the bandwidth meaning? Or is it how much data can fit, which is the capacity meaning? The same two words can point to two different constraints, so confirm the speaker's meaning before acting on it.

Inference costs are also tied to the economics of the model itself. The source explains that inference involves the forward pass, while training involves both the forward and backward passes. It then describes a relationship between training cost and inference revenue from the viewpoint of a model provider. Using the variables described in this relationship, the condition is written as follows.