Integrating inference techniques into a cohesive system

Training a model is only part of its story. Once a trained model starts answering requests in production, its speed and its cost depend on how the serving system is organized, as well as on the model itself. Two machines running the same model can deliver very different experiences depending on how requests reach them and how work is spread across hardware. This article follows one thread through that problem: how the different inference techniques fit together into a single serving system, and how to reason about the trade-offs they create.

Follow the state of the system step by step. In the first state, a single machine serves the model and receives every request. All of the work for every request lands on that one machine, so each request competes with the others for the same resources.

In the second state, you decide to run more replicas of the model on additional machines to lower response times. Traffic is now divided among the machines, so each machine receives fewer requests than it did before. The source does not describe the routing mechanism that splits traffic among replicas, so treat that routing step as a placeholder for whatever distributes requests in a given serving system. What the source does establish is the effect: with more replicas sharing the same load, each machine handles fewer requests.

Inference optimization has two families. The first family works on the model. Quantization and distillation are model-agnostic techniques, meaning they can be applied across different model architectures. Other techniques are tailored to architecture. Because the attention mechanism is a key bottleneck in transformer models, many optimizations target attention directly, including KV cache management and writing attention kernels.

The second family works at the level of the inference service, meaning how requests are grouped and how work is spread across hardware. This family includes batching and parallelism strategies. Parallelism strategies are the subject of this article.

Keep the two families separate in your mind. Model-level techniques change what the model computes or how cheaply it stores its weights and attention state. Service-level techniques change how many requests are handled at once and where they run. The source is explicit that no single technique is always the right one. The choice depends on the workload and on the performance requirements. Among the techniques the source singles out as most impactful across use cases are quantization, tensor parallelism, replica parallelism, and attention mechanism optimization. Two of those four, tensor parallelism and replica parallelism, are the parallelism strategies this article examines.

Replica parallelism is the simplest parallelism strategy to picture. You run more copies of the model on more machines so that each machine handles fewer requests. The source ties its cost to a clear trade-off: scaling up replica parallelism costs more machines, but it buys lower latency. The source also describes it as relatively straightforward to implement, which is one reason it appears among the most impactful techniques.

Tensor parallelism is the second strategy the source ranks among the most impactful, and it is framed differently. The source gives it two named benefits: it both reduces latency and enables serving larger models. Replica parallelism is about spreading requests across identical copies. The source's description of tensor parallelism is framed around the model itself and the latency of a response, not around spreading whole requests across copies. The source does not describe how tensor parallelism divides the work internally, so this article does not explain that mechanism. What you should take away is the pair of effects it is credited with: a model too large for a single machine can be served at all, and the response can arrive faster.

Before you can discuss bottlenecks, you need to clear up a term that appears constantly in inference discussions: memory-bound. A common assumption is that this phrase has one fixed meaning, so any engineer who says a workload is memory-bound means the same thing as any other. The source reports that this assumption does not hold. The word is used with different meanings depending on the speaker's background.

Trace the meaning through two teams. A systems team, such as optimization engineers and GPU engineers, tends to use memory-bound to mean bandwidth-bound: the limit is how fast data can move. An AI team, such as ML and AI engineers, tends to use it to mean memory capacity-bound: the limit is how much data can fit. The Roofline paper uses the term to mean memory-bandwidth bound, which matches the systems usage.

The practical habit this suggests is to ask one question whenever you read or hear the phrase: which resource is the limit? If the answer is how fast data can move, the speaker probably means bandwidth. If the answer is how much data can fit, the speaker probably means capacity. The source says to confirm which meaning a speaker intends before acting on it, because a fix for one kind of bottleneck may do nothing for the other.

Inference costs are also tied to the economics of the model itself. The source explains that inference involves only the forward pass, while training involves both the forward and backward passes. It then describes a relationship between training cost and inference cost from the viewpoint of a model provider. Reading the notation used in this relationship, $T$ stands for training cost, $N$ for the number of inference calls, and $p$ for the amount recovered per call.

The source reads this inequality as a condition for developing a model to make sense at all: the money recovered from inference must exceed the training cost. Read it as a question about thresholds. Holding the recovery per call fixed, the number of calls $N$ must grow large enough that $p \times N$ reaches $T$. Any region where $T$ is greater than $p \times N$ is a region where the provider has not yet recovered the cost of training. The source also observes that the more a model is used in production, the more providers can reduce inference cost.