Model Inference Basics

Imagine typing a question into an AI assistant and watching the cursor sit still for several seconds before the first word appears. Most people would call that experience slow, and many would stop using the product. The delay is not a cosmetic problem. It is the visible result of how long a trained model takes to do its work when someone asks it something.

Inference optimization is the set of efforts aimed at shrinking that delay and the cost of producing answers, so that models respond faster and run more cheaply. This article explains why that goal matters, why it has grown more urgent as models have become larger, and how the sequential way many of these models generate output makes it difficult to achieve.

The two goals serve two different audiences. Speed matters to the people using a product, because a faster model gives them a response sooner. Cost matters to the organization running the product, because cheaper inference lowers what it spends to answer each request. The source puts it bluntly: users never say no to faster models, and companies can always benefit from cheaper inference.

Neither goal is ever fully finished. That is why optimizing inference has been important for as long as machine learning engineering has existed. The pressure has grown, however, with the arrival of foundation models. As these models scale up, the cost of each inference rises, and so does latency, which is the time between a request and its response. A much larger model does more work per output, so an optimization effort that was once sufficient now has much more at stake. This is why inference optimization has become an active subfield in both industry and academic research.

Many foundation models are autoregressive, which means they generate tokens sequentially. A token is a unit of output the model produces one at a time. The model cannot produce the whole answer in a single shot. It produces one token, uses that work to produce the next, and continues until the output is complete.

Trace the loop in the diagram below. The prompt goes in, and the model writes Token 1. To write Token 2, the model needs the work from Token 1, so it cannot start Token 2 early. Token 3 depends on the tokens before it in the same way. The defining feature is ordering: each step depends on the steps before it. The work cannot simply be spread out and run all at once, the way a set of independent calculations might be.

Each pass through this loop takes time, and the number of passes equals the number of tokens in the output. This is the reason autoregressive generation is costly. A longer answer means more passes, and each pass adds its own delay.

Because the number of passes equals the number of output tokens, the total delay is the per-token time multiplied by the length of the output. If t stands for the time to generate one token and n stands for the number of output tokens, the approximate total response time is given by the formula below.

The formula shows why length matters so much. Doubling the number of output tokens doubles the total delay, if the per-token time stays the same. A response with many tokens accumulates delay with every step, so even modest per-token times can push the whole response well beyond the benchmark discussed later in this article.

The timeline above uses an illustrative per-token time of 50 ms. It is not a measurement of any particular model. The point is the pattern: each new token adds one more step to the clock, and the clock never resets between tokens.

Latency is the time an AI application takes to respond. It is one of the main ways a user experiences a system's speed, and it is the figure that inference optimization tries to reduce. The source points to a specific benchmark: a typical internet application is expected to respond within about 100 ms. Users of such applications have learned to expect that speed, and they are notoriously impatient with anything slower.

Getting AI applications down to that roughly 100 ms expectation is a major challenge. The reason connects back to autoregressive generation. A typical web request is a short exchange with a fixed, small amount of work. An AI response that contains many tokens, each produced in sequence, has many steps to pass through before it is finished. The gap between what users expect and what sequential generation naturally delivers is what makes this a central problem for the field, and it is why the area has become an active research topic.