Batching in Inference

Once a model has been trained, applications send it inputs and wait for outputs. Many model providers expose this work through two kinds of inference API: an online API and a batch API. Both accept requests and both return predictions, so from the outside they can look alike. The difference lies in what the provider is trying to optimize. That choice shapes how requests are scheduled, how the hardware is used, and what the person waiting on the result experiences.

This article follows a small set of requests through each API style. As you read, keep one question in mind: when a request arrives, what is the provider allowed to do with it before answering?

Suppose three requests arrive close together: a chatbot message, a code-generation request, and a request to summarize a day's support tickets. The scenario is illustrative and was written to make the two behaviors concrete. Each request is a separate job for the provider.

Read the online row first. Each request is handled as soon as it arrives, because the online API is tuned for the time a single caller waits. The chatbot message is not held up by the summary job that arrives alongside it. Now read the batch row. Its requests do not need an immediate answer, so the provider is free to hold them, group many of them together, and process them as one unit of work. In this scenario the three requests wait in the queue rather than being answered on arrival, and each one finishes when the provider completes the work it belongs to. The diagram does not give finish times, because the source does not specify them for batch jobs. What it does show is the difference in when processing begins.

An online API is tuned for latency, which means the time a single caller waits for an answer. Each request is handled as soon as it arrives. A batch API is tuned for cost. Its requests do not need to be answered immediately, and that slack gives the provider room to use cheaper hardware and to group many requests together so they are processed as one unit of work.

The source describes the batch trade-off in concrete terms. When turnaround time stretches from seconds or minutes to hours, the provider can apply a wider range of optimization techniques. At the time the source was written, Google Gemini and OpenAI both offered batch APIs at a 50% cost reduction in exchange for that longer turnaround.

It would be a mistake to conclude that online APIs never batch anything. The source notes that online APIs may still group requests together, provided the grouping does not significantly hurt latency. The deciding factor is therefore not whether grouping happens at all, but how much waiting it adds. The real difference is one of emphasis: an online API focuses on keeping latency low, whereas a batch API focuses on maximizing throughput.

The pipeline shows the general pattern of grouping. Separate requests are collected, the group is run through the model as one unit of work, and each caller still receives its own result. The source confirms that a batch API groups requests so they are processed together and that both API styles return predictions. It does not describe internal scheduling details, so the diagram should be read as a picture of the idea rather than of any particular provider's implementation.

Language models often produce output one token at a time through autoregressive decoding, and generating a full response can take a long time. Users, meanwhile, are impatient. APIs usually return a complete response by default, but many online APIs offer streaming mode, which returns each token as soon as it is generated.

The sequence diagram shows the client sending one prompt and then receiving a series of messages, each carrying a token, rather than one message containing everything. The client can start displaying text as soon as the first token arrives. The main benefit is that the time a user waits until the first token shrinks, even though the complete answer still takes just as long to finish.

Streaming buys responsiveness at a cost in safety checking. If the whole response is generated before anything is shown, the application can score that response and decide whether it is acceptable. When tokens are streamed, that step is skipped, because the text reaches the user before it is complete. The source states this plainly: you cannot score a response before showing it to users, which raises the risk that a bad response is seen.

The word batch means something slightly different in traditional machine learning. There, online inference means predictions are computed after a request has arrived, while batch inference means predictions are precomputed before any request arrives. The source separates this meaning from the batch API described earlier, because the two are not the same thing for foundation models.

Precomputation works when inputs are finite and predictable. A recommendation system, for example, can generate recommendations for every user in advance, store them, and retrieve them when a user visits the site. Foundation model inputs are open-ended, and it is hard to anticipate every prompt a user might write. Precomputing answers for all of them is therefore impractical, which is why the batch API for foundation models works by processing submitted requests rather than by precomputing predictions.

Before optimizing an inference service, it helps to know which measurements matter. The source identifies three: latency, throughput, and utilization. From the user's point of view, latency is the central concern. The source also makes a boundary point: response quality is a property of the model itself, not of the inference service that delivers it, so an inference API cannot be judged on quality alone.

Application developers must look further than latency. Latency tells you how a single caller experiences the service, while throughput and utilization tell you what the service is doing in aggregate. Both views are needed to decide whether an online or batch design makes sense. The source does not give numeric relationships between these metrics, so it is best to treat them as complementary views rather than as values that can be read from one another.

The source also notes that grouping has a visible cost on latency when it adds waiting. This is why the question of how much waiting a grouping step introduces matters so much when choosing a design.

The source lists workloads that suit batch APIs because they tolerate delay. These include generating synthetic data, producing periodic reports such as summaries of Slack messages or analyses of customer support tickets, processing every document uploaded during onboarding of a new customer, reprocessing all data when migrating to a new model, generating personalized recommendations or newsletters for a large customer base, and reindexing an organization's data to update a knowledge base. What these share is that no individual caller needs an answer right away.