Evaluating Model APIs

Imagine two AI assistants built on the same underlying model. The first returns its answer after a noticeable pause, and each word arrives slowly. The second starts answering almost immediately and keeps going at a steady pace. The content of both answers may be identical, yet a user will probably find the second assistant more pleasant to work with. The difference lies in how efficiently the model's inference runs, and that difference shapes which applications can reasonably be built on top of the model.

The question this article answers is how to measure that difference. Before you can judge whether one model API is efficient, or whether an optimization helped, you need to know what you are measuring. The sections below follow a single response through inference, separate the main efficiency metrics, and then look at the trade-offs and the layers where optimization happens.

A model's usability depends heavily on its inference cost and latency. Inference cost is what it takes to run the model for a request. Latency is how long the user waits. These two properties decide what can be built with a model. Cheaper inference makes AI-powered decisions more affordable, and faster inference makes it possible to place AI in a wider range of applications.

Cost and latency are not separate concerns that happen to sit side by side. A model that is accurate but too slow for a conversation, or too expensive to run on every request, may still be impractical to deploy. Usefulness is therefore a property of the model together with the way it is run, not of the model's answers alone.

The source groups the common efficiency metrics into three families: latency, throughput, and utilization. Each one answers a different question about the same running system. Latency asks how long a single request takes to get a response. Throughput asks how much work the model can process over a period of time. Utilization asks how much of the available hardware capacity is being put to use. Keeping these questions separate matters because a system can look excellent on one of them and poor on another.

The three metrics are not interchangeable. A request can have a short wait while the system as a whole handles only a modest amount of work, or it can handle a large amount of work while each request waits longer. Reading one metric alone gives an incomplete picture of the system.

For language model-based inference, the source recommends looking inside latency rather than treating it as one number. Latency can be split into two parts. One clock measures how long you wait before anything appears. The other measures how quickly the rest of the output follows. The diagram below traces one response to show where each clock starts and stops.

Think about a single response being generated one token at a time. Before the first token appears, the model has to take in the input, which the source ties to the prefilling phase. Only after that does the model produce the rest of the output, which the source ties to the decoding phase. The two latency clocks are therefore not interchangeable. One is paid at the start of a response, and the other is paid again for every token that follows.

Throughput is the lens through which cost is most directly seen. The source states that throughput metrics are directly related to cost. The more inputs a system can process in a given period, the more work each unit of hardware and time is doing. When throughput is low, the same hardware must stay busy for longer to handle the same workload, and that expense accumulates.

The more important lesson is the trade-off the source names between latency and throughput. Reducing latency often increases cost, and you can potentially reduce cost if you are willing to accept higher latency. The two metrics pull against each other. The diagram below sets the two tunings side by side.

Neither end of this trade-off is automatically correct. A system tuned to answer each request as quickly as possible often gives up some ability to handle many requests at once. A system tuned to handle a large volume of work may make each individual request wait longer. The right balance depends on what the application needs.

All of these metrics depend on the physical machine doing the work. The source is direct about this: how efficiently a model can run depends on the hardware it runs on. The same model can therefore show different latency and throughput numbers on different hardware. The source also notes that optimizing models on different accelerators is important. It gives only this high-level point, so the key idea for this article is that hardware is part of the efficiency picture and cannot be ignored when comparing numbers.

Once you can measure efficiency, the next question is how to improve it. The source describes inference optimization techniques operating at two levels: the model level and the inference service level. The distinction is less about speed and more about what is allowed to change. Model-level optimization often requires changing the model itself, and the source warns that this can lead to changes in model behaviors. Inference service-level optimization typically keeps the model intact and changes only how the model is served.

The layers answer different questions. A model-level change asks what the model should be. A service-level change asks how a fixed model should be run for the people using it. When a provider describes an optimization, knowing which layer it sits on tells you whether the model's behavior may have changed.

The source makes a practical point: most application developers will use model APIs with the optimizations already built in, rather than implementing these techniques themselves. You do not need to build an optimization stack to evaluate one. Understanding what is possible gives you better questions to ask when a provider quotes a fast or cheap service.