Autoregressive Decoding and Token Generation

Once a model has been trained, applications send it inputs and wait for outputs. Many model providers expose this work through two kinds of inference API: an online API and a batch API. Both accept requests and both return predictions, so from the outside they can look alike. The difference lies in what the provider is trying to optimize. That choice shapes how requests are scheduled, how the hardware is used, and what the person waiting on the result experiences.

Language models often produce their output one token at a time through autoregressive decoding. A token is one piece of the output, and the model does not emit a whole response in a single step. Each step produces the next token, and that token becomes part of what the model works from when it produces the step after it. The loop repeats until the response is complete.

Because the response is built this way, generating a full answer can take a long time. The diagram below traces the loop: the text so far is used to predict the next token, that token is appended, and the extended text becomes the input for the next prediction.

An online API is tuned for latency, meaning the time a single caller waits for an answer. Each request is handled as soon as it arrives. A batch API is tuned for cost. Its requests do not need to be answered immediately, and that slack gives the provider room to use cheaper hardware and to group many requests together so they are processed as one unit of work.

A useful way to hold this distinction is to ask what the provider is allowed to delay. An online API cannot delay much, so it gives up some efficiency in order to answer quickly. A batch API may delay, so it can spend that time on efficiency. The source describes the batch trade-off concretely: when turnaround stretches from seconds or minutes to hours, the provider can apply a wider range of optimization techniques. When the source was written, Google Gemini and OpenAI both offered batch APIs at a 50% cost reduction in exchange for that longer turnaround.

The clearest way to see the difference is to follow requests through the system. Suppose three requests arrive close together: a chatbot message, a code-generation request, and a request to summarize a day's support tickets. Each request is a separate job for the provider. The timeline below shows how an online API and a batch API might handle them. Each lane is read from left to right in order of time.

The word batch means something slightly different in traditional machine learning. There, online inference means predictions are computed after a request has arrived, while batch inference means predictions are precomputed before any request arrives. The batch API described above is not the same thing, and the distinction matters for foundation models.

Precomputation works when inputs are finite and predictable. A recommendation system, for example, can generate recommendations for every user in advance, store them, and retrieve them when a user visits the site. Foundation model inputs are open-ended, and it is hard to anticipate every prompt a user might write. Precomputing answers for all of them is therefore impractical, which is why the batch API for foundation models works by processing submitted requests instead.

Each workload in that list tolerates delay. What they share is that no individual caller needs an answer right away, and that is the condition under which a batch API's cost advantage is available.

APIs usually return a complete response by default. Many online APIs also offer streaming mode, which returns each token as soon as it is generated. Because users are impatient and generating a full response takes time, streaming changes what the user experiences. The diagram shows the client sending one prompt and then receiving a series of messages, each carrying a token, rather than one message containing everything. The client can begin displaying text as soon as the first token arrives.

Before optimizing an inference service, it helps to know which measurements matter. The source identifies three: latency, throughput, and utilization. From the user's point of view, latency is the central concern, because it describes how long a single caller waits. Throughput and utilization describe the service in aggregate: how much work it completes and how efficiently it uses its resources.

Response quality is a further consideration, but the source makes a boundary point about it: quality is a property of the model itself, not of the inference service that delivers it. An inference API therefore cannot be judged on quality alone. Latency tells you how one caller experiences the service, while throughput and utilization tell you what the service is doing overall. Both views are needed to decide whether an online or batch design makes sense.