Measuring Inference Latency and Throughput

Once a model has been trained, applications send it inputs and wait for outputs. Many model providers expose this work through two kinds of inference API: an online API and a batch API. Both accept requests and both return predictions, so from the outside they can look alike. The difference lies in what the provider is trying to optimize, and that choice shapes how requests are scheduled, how the hardware is used, and what the person waiting on the result experiences.

An online API is tuned for latency, which means the time a single caller waits for an answer. Each request is handled as soon as it arrives. A batch API is tuned for cost. Its requests do not need to be answered immediately, and that slack gives the provider room to use cheaper hardware and to group many requests together so they are processed as one unit of work.

A helpful way to hold this distinction is to ask what the provider is allowed to delay. An online API is not allowed to delay much, so it gives up some efficiency to answer quickly. A batch API is allowed to delay, so it can spend that time on efficiency. The provider's documentation describes the batch trade-off in concrete terms: when turnaround time stretches from seconds or minutes to hours, the provider can apply a wider range of optimization techniques. At the time that material was written, Google Gemini and OpenAI both offered batch APIs at a 50% cost reduction in exchange for that longer turnaround.

The clearest way to see the difference is to follow a small set of requests through each style. Suppose three requests arrive close together: a chatbot message, a code-generation request, and a request to summarize a day's support tickets. Each request is a separate job for the provider. The scenario below is an illustration written to make the two behaviors concrete.

It would be a mistake to conclude that online APIs never batch anything. Online APIs may still group requests together, provided the grouping does not significantly hurt latency. The deciding factor is therefore not whether grouping happens at all, but how much waiting it adds. The real difference is one of emphasis: an online API focuses on keeping latency low, whereas a batch API focuses on maximizing throughput.

The word batch means something slightly different in traditional machine learning. There, online inference means predictions are computed after a request has arrived, while batch inference means predictions are precomputed before any request arrives. This meaning is not the same as the batch API described above, because the two are not interchangeable for foundation models.

Precomputation works when inputs are finite and predictable. A recommendation system, for example, can generate recommendations for every user in advance, store them, and retrieve them when a user visits the site. Foundation model inputs are open-ended, and it is hard to anticipate every prompt a user might write. Precomputing answers for all of them is therefore impractical, which is why the batch API for foundation models works by processing submitted requests rather than by precomputing predictions.

These workloads share one property: no individual caller needs an answer right away. That property is what makes a batch API a good fit.

Language models often produce output one token at a time through autoregressive decoding, and generating a full response can take a long time. Users are impatient. By default, APIs usually return a complete response, but many online APIs offer streaming mode, which returns each token as soon as it is generated.

The main benefit is that the time a user waits until the first token shrinks. The client can start displaying text as soon as the first token arrives. The complete answer still takes just as long to finish, so streaming improves responsiveness rather than total generation time.

The main drawback is a loss of safety checking. If the whole response is generated before anything is shown, the application can score that response and decide whether it is acceptable. When tokens are streamed, that step is skipped, because text reaches the user before it is complete. In short, a response cannot be scored before it is shown to users, which raises the risk that a bad response is seen.

Before optimizing an inference service, it helps to know which measurements matter. Three are central: latency, throughput, and utilization. From the user's point of view, latency is the central concern. Latency is the time a single request takes, the wait that one caller experiences. Throughput is the number of requests the service completes per second. Utilization is the share of the hardware that is in use.

Response quality is a property of the model itself, not of the inference service that delivers it. An inference API therefore cannot be judged on quality alone. Application developers must look at throughput and utilization as well, because these shape how much work the service can do and how efficiently it uses its resources, and those in turn determine what the application costs to run.