Inference Optimization Techniques

Imagine two AI assistants built on the same underlying model. The first pauses noticeably before it answers, and then each word arrives slowly. The second starts answering almost immediately and keeps going at a steady pace. Even if both give identical content, most people would find the second one more pleasant to work with. The difference comes from how efficiently the model's inference runs, and that difference shapes which applications can reasonably be built on top of the model.

Once a model exists, it is used to produce outputs. That use is called inference. Inference optimization is the set of techniques that aim to make this use of a model better optimized. A model's usability depends heavily on its inference cost and latency. Cheaper inference makes AI-powered decisions more affordable, and faster inference makes it possible to put AI into a wider range of applications. Because so much value depends on these two properties, it is worth learning how they are measured before trying to improve them. You cannot tell whether an optimization helped unless you know what you were measuring in the first place.

Keep one question in mind as you read: if several different techniques can each optimize inference, how do they relate to one another? This overview answers that question at the level of names and roles. The detailed mechanics are covered in later chapters.

The common efficiency metrics fall into three families: latency, throughput, and utilization. Each one answers a different question about the same running system. Latency asks how long a single request takes to get a response. Throughput asks how much work the model can process over a period of time. Utilization asks how much of the available hardware capacity is being put to use. Keeping these questions separate is the first step, because a system can look excellent on one measure and poor on another.

For language model-based inference, the source recommends looking inside latency rather than treating it as one number. Latency can be broken into two parts. The first is time to first token, abbreviated TTFT, which is influenced by the prefilling phase. The second is time per output token, abbreviated TPOT, which is influenced by the decoding phase. Put simply, one clock measures how long you wait before anything appears, and the other measures how quickly the rest of the output follows.

Throughput is the lens through which cost is most directly seen. The source states that throughput metrics are directly related to cost. The more inputs a system can process in a given period, the more work each unit of hardware and time is doing. When throughput is low, the same hardware must stay busy for longer to handle the same workload, and that expense accumulates.

The more important lesson is the trade-off the source names between latency and throughput. You can potentially reduce cost if you are willing to accept increased latency, and reducing latency often involves increasing cost. The two metrics pull against each other. A system tuned to answer each request as quickly as possible often gives up some of its ability to handle many requests at once. A system tuned to handle a large volume of work may make each individual request wait longer. Neither end of this trade-off is automatically correct. The right balance depends on what the application needs.

Once you can measure efficiency, the next question is how to improve it. The source describes inference optimization techniques that operate at two levels: the model level and the inference service level. The distinction is less about speed and more about what is allowed to change. Model-level optimization often requires changing the model itself, and the source warns that this can lead to changes in model behaviors. Inference service-level optimization, by contrast, typically keeps the model intact and changes only how the model is served.

The source also points out a practical reality. Given the availability of model APIs, most application developers will use those APIs with their built-in optimization instead of implementing these techniques themselves. The techniques may not be relevant to every application developer, but understanding what is possible helps you evaluate the efficiency of a model API you are considering. When a provider quotes a fast or cheap service, knowing which level of optimization might be behind that claim gives you better questions to ask.

Each technique is introduced here only by name and role. Quantization, distillation, and parallelism are discussed in detail in Chapters 7 through 9. Treat this section as a map. It tells you what each technique is called, and the later chapters supply the mechanics.

This overview introduces the inference optimization material in the module. The source says the three techniques are discussed in Chapters 7 through 9, so the detailed mechanics of quantization, distillation, and parallelism are found in those chapters rather than in this overview.

Application development is the topic that follows the inference optimization material. The source presents the two in sequence, with its heading appearing directly after the optimization material. The order is the lesson to carry forward: the optimization techniques come first, and application development comes next. The source does not describe how application development depends on these techniques, so do not assume a specific link beyond that order.