Latency Metrics for Token Generation

When people compare hardware for language models, the first number they look at is usually peak FLOP/s, the maximum floating-point operations a chip can perform each second. The reasoning seems sound: more operations per second should mean more speed. This lesson shows why that reasoning fails for serving, and what to measure instead. The two measures that matter most for token generation are the efficiency of the compute your workload really uses, and the delay a user perceives between tokens.

The formula makes the key point visible. A chip with a huge peak number can still produce low MFU if the workload cannot keep that hardware busy. The peak number belongs to the chip under chosen conditions. MFU belongs to your model running your workload. Two systems on the same chip can therefore show very different MFU values.

Most of the arithmetic in a neural network is matrix multiplication, usually called matmul. Estimates cited in the source put matmul at more than 90 percent of all floating-point operations in a neural network. That is why optimization work tends to focus on how well matmul uses the hardware. If matmul runs efficiently, MFU is high. If matmul stalls on data movement, MFU drops even when the chip is powerful.

Hardware and model design also shape each other. A chip can be built to run one architecture well, and an architecture can be designed to make the most of a chip. The transformer, for example, was originally designed by Google to run fast on TPUs and was only later optimized for GPUs. Memory technology matters too: lower-end and mid-range GPUs may use GDDR (Graphics Double Data Rate) memory, which affects how quickly data can reach the compute units.

Token generation latency is measured with two related metrics. Time between tokens (TBT) is the name used by LinkedIn, and inter-token latency (ITL) is the name used by NVIDIA. Both describe the delay between consecutive generated tokens, which is what a user perceives as the model typing. When you compare reported numbers, check which metric a team means.

Not every delay has the same effect. An experiment by Anyscale showed that 100 input tokens have roughly the same impact on overall latency as a single output token. A long prompt is therefore not free just because it produces little output. User expectations also shift over time: in the early days of chatbots, some people complained that responses came too fast and seemed unnatural, but as people became familiar with chatbots, that stopped being the case.

Batching for inference starts with a service decision: split the inference API into an online path and a batch path. Online requests are the ones where latency matters most, such as a user waiting on a response. Batch requests are less urgent and can be sent to a batch API and processed without the same immediacy. Separating the two lets the service focus its capacity on online requests first, so a large batch job does not delay a person waiting at a chat window.

Caching in inference means saving work that would otherwise be repeated. For transformer models, prefilling populates the initial KV cache, which stores the key and value data the model computes for tokens it has already processed. Prefilling fills this cache before generation proceeds, so later steps can reuse the stored work instead of recomputing it from scratch.

The reason prefilling pays off is that the system prompt of an application is usually known in advance. Developers write it once and it repeats across many requests. The exact user queries are the hard-to-predict part. The stable portion of each request can therefore be processed ahead of time and kept in the cache, while only the unpredictable portion is handled fresh for each request.

Read the diagram left to right. The system prompt tokens sit in the cache with their stored key and value data, so a later request that begins with the same system prompt does not need that work repeated. Only the user query tokens, the unpredictable part, are computed fresh. The stored state is reused during the decode phase, which is how earlier tokens avoid being recomputed.

Prompt caching applies the same idea at the level of prompts: if a prompt has been processed before, store the result and reuse it rather than recompute it. Implementations vary a great deal, so confirm whether a given system caches whole prompts, prefixes, or only earlier messages in a session. One example from the source is llama.cpp, which has prompt caching but appears to cache only whole prompts, and only for queries within the same chat session. Caching whole prompts means a new prompt that merely shares a prefix with an old one may not benefit.

Token generation is bandwidth-heavy. Each generation step requires transferring the entire model's parameters from the accelerator's high-bandwidth memory to its compute units. Because the model produces only one token at a time, that step uses only a small number of FLOP/s, which is a form of computational inefficiency. Speculative decoding is the technique the source names as addressing this situation.

The commonly described pattern is this: a smaller draft model proposes several tokens ahead, and the main target model checks them together in one pass, keeping the proposals it accepts. The source pack gives only this motivation and not the full acceptance mechanism, so treat the steps below as the general pattern rather than a detail the source specifies.

The limits of speculative decoding follow from its motivation. It targets the bandwidth-bound, one-token-at-a-time step. If a workload is not dominated by that step, the gain it targets is smaller. The source pack does not enumerate other specific limits, so for any given serving stack, measure MFU and token-to-token latency before and after enabling it.