How Foundation Models Generate Output

Think about the last time you waited for a chatbot to finish an answer. The first words may have appeared quickly, and then the rest trickled in while you watched. That pacing is not a cosmetic choice. It reflects how many modern foundation models produce their output, and it is the starting point for one of the central difficulties in deploying them.

The question this article answers is simple to state: if a model generates its output in pieces, what does that do to the time a user spends waiting, and what can engineers do about it? To answer it, we first need to look closely at the mechanism that produces the output.

The word sequentially is the key to this whole topic. Generation is not a single operation that returns a whole answer at once. The model produces one token, then another, and the sequence of those steps is what determines how long the full output takes. Many foundation models work this way, and the source identifies this step-by-step production as one of the challenges with them.

Models of this kind are called autoregressive. Each new token is produced by its own step, and each step is part of a loop. The diagram below shows the shape of that loop. It is a picture of the process rather than of any particular model's internals, so the boxes describe what happens in general, not how one specific system is built.

Read the loop from top to bottom. The model looks at what it has produced so far, generates the next token, adds that token to the output, and checks whether the output is finished. If it is not finished, the loop runs again. Each pass through the loop is one step, and the total number of steps is the number of tokens in the answer. This is why the cost of one step matters so much: it is multiplied by every token the model emits.

The source gives a simple way to reason about the cost of generation. Suppose a model takes a fixed amount of time to generate one token. Because the tokens are produced one after another, the total time grows with the number of tokens. The source's own example is a model that takes 10 ms per token: a 100-token output takes about one second, and longer outputs take even longer.

The source expresses this as an approximation. Let T be the total generation time, n the number of tokens, and t the time needed to generate one token. Then:

The approximation ignores overhead that falls outside the per-token work, so it is an estimate rather than an exact prediction. It still captures the central point. When each token costs a fixed amount of time, the total wait is proportional to the output length. Doubling the number of tokens roughly doubles the wait.

The timeline shows the build-up. Each token adds the same 10 ms to the running total, so the elapsed time climbs steadily rather than jumping at once. By the hundredth token, the running total has reached about one second.

Notice what the worked example does not need. It does not need to know anything about the internal design of the model. Once the per-token time is known, the estimate follows directly from the number of tokens. That is what makes the approximation useful for planning: an engineer can estimate how a change in output length will change the wait.

Latency is the delay in generating output. The source notes that users are notoriously impatient, and it sets a benchmark: about 100 ms of latency is expected for a typical internet application. Getting AI applications down to that level is described as a huge challenge.

The diagram below traces the request path from the user's side. The user waits at the point where the model is producing tokens, and the application passes those tokens along as they are generated.

Read the sequence in order. The user sends a prompt, the application forwards it, and the model returns tokens one at a time. The wait the user feels is shaped by the model's per-token steps: the longer the output, the longer the model keeps producing tokens. This is why the trickling pace you noticed in a chatbot is connected to the time the model needs to finish.

Inference refers to the work of running a trained model to produce output. Inference optimization is the discipline of making that work faster and cheaper. The source frames it as always important in machine learning engineering, because users never say no to faster models and companies always benefit from cheaper inference.

As foundation models push inference cost and latency higher, inference optimization has become even more important, and it is now an active subfield in both industry and academia. The two ideas connect directly. Autoregressive generation is one of the reasons latency is hard to reduce, and inference optimization is the work devoted to reducing it. The challenge described in this article is the problem that motivates that work.

The source does not describe specific optimization techniques, so this article does not claim how any particular method works. What you can take away is the logic of the link: since total time accumulates with the number of tokens and the per-token cost, any progress on making generation faster or cheaper is directly relevant to the wait a user experiences.