Hardware Considerations for AI Models

Imagine two AI assistants built on the same underlying model. The first returns its answer after a noticeable pause, and each word arrives slowly. The second starts answering almost immediately and keeps going at a steady pace. A user will probably judge the second one as more pleasant to work with, even if the content is identical. The difference lies in how efficiently the model's inference runs, and that difference shapes what applications can reasonably be built on top of the model.

Inference efficiency matters because a model's usability depends heavily on its inference cost and latency. Cheaper inference makes AI-powered decisions more affordable, and faster inference makes it possible to put AI into a wider range of applications. Because so much value depends on these two properties, it is worth learning how they are measured before trying to improve them. You cannot tell whether an optimization helped unless you know what you were measuring in the first place.

Think about a single response being generated, one token at a time. Before the first token appears, the model has to take in the input, and the source ties this step to the prefilling phase. Only after that does the model produce the rest of the output, which the source ties to the decoding phase. The diagram below traces this order for one request. Read it from left to right and notice where the first wait ends and where the repeated gaps begin.

The source groups the common efficiency metrics into three families: latency, throughput, and utilization. Each one answers a different question about the same running system. Latency asks how long a single request takes to get a response. Throughput asks how much work the model can process over a period of time. Utilization asks how much of the available hardware capacity is being put to use. Keeping these questions separate matters, because a system can look excellent on one of them and poor on another.

For language model-based inference, the source recommends looking inside latency rather than treating it as one number. Latency can be broken into two parts. The first is time to first token, abbreviated TTFT, which is influenced by the prefilling phase. The second is time per output token, abbreviated TPOT, which is influenced by the decoding phase. Put simply, one clock measures how long you wait before anything appears, and the other measures how quickly the rest of the output follows.

Because of this split, the two latency clocks are not interchangeable. TTFT is paid once, at the start of a response. TPOT is paid again for every token that follows. A system can have a short TTFT and still feel slow if its TPOT is large, and the reverse is also possible. Measuring only the total time hides which part needs work.

Throughput is the lens through which cost is most directly seen. The source states that throughput metrics are directly related to cost: the more inputs a system can process in a given period, the more work each unit of hardware and time is doing. When throughput is low, the same hardware must stay busy for longer to handle the same workload, and that expense accumulates.

The more important lesson is the trade-off the source names between latency and throughput. You can potentially reduce cost if you are willing to accept increased latency, and reducing latency often involves increasing cost. The diagram below shows the two sides of this choice when a server groups more requests together. Grouping raises the number of requests served per period, but each individual request may wait longer.

Once you can measure efficiency, the next question is how to improve it. The source describes techniques for inference optimization that operate at two levels: the model level and the inference service level. The distinction is less about speed and more about what is allowed to change. The hierarchy below separates the two levels by what each one touches.

The two levels are easy to confuse because both aim at the same goals of lower cost and lower latency. The difference is the object being changed. Model-level work can alter the model, and the source warns that this can lead to changes in model behaviors. Service-level work typically keeps the model intact and changes how requests are handled, including how many are grouped together, which is the same lever behind the latency and throughput trade-off above.

The source also points out a practical reality. Given the availability of model APIs, most application developers will use these APIs with their built-in optimization instead of implementing these techniques themselves. Understanding both levels still helps you evaluate a model API you are considering. When a provider quotes a fast or cheap service, knowing which level of optimization might be behind that claim gives you better questions to ask.

All of these metrics depend on the physical machine doing the work. How efficiently a model can run depends on the hardware it is run on, so the same model can show different latency and throughput numbers on different hardware. The source also notes that optimizing models on different accelerators is important. It gives this only as a high-level point, so the key idea for now is that hardware is part of the efficiency picture and cannot be ignored when comparing numbers.