Model FLOPS Utilization and Hardware Choice

Suppose one chip advertises a higher peak FLOP/s number than another. It is tempting to conclude that a model running on the first chip will answer requests faster. The reasoning sounds sensible: FLOP/s counts floating-point operations per second, so more operations per second should mean more speed. Inference optimization is largely about discovering where this assumption breaks down and deciding what to measure instead.

The rest of this article follows that question through the serving stack. It starts with a measure of how much of a chip's capacity a workload actually uses, then traces how a service splits its traffic, how cached work is stored and reused, and where those shortcuts stop helping.

The idea of tracking FLOP/s utilization has been around for a long time, but the term MFU was introduced in the PaLM paper by Chowdhery et al. in 2022. Because MFU is a ratio of achieved work to available capacity, it can be written as the following relationship, where achieved FLOP/s is the rate of arithmetic your workload really performs and peak FLOP/s is the chip's stated ceiling.

The formula explains the puzzle from the opening. A chip with a very large peak number sits in the denominator, so if the workload cannot keep the hardware busy, the achieved FLOP/s stays small relative to that peak and MFU is low. A large denominator does not help a workload that never reaches it.

The gap between peak and real-world numbers has a second source. Chip makers can measure peak FLOP/s under special conditions, such as using sparse matrices with specific shapes. Those conditions can raise the peak figure without describing the workloads most users actually run. So the peak number is a property of the chip under chosen conditions, while MFU is a property of your model on your workload.

Most of the arithmetic in a neural network is matrix multiplication, usually called matmul. Estimates cited in the source put matmul at more than 90% of all floating-point operations in a neural network. This is why optimization work tends to focus on how well matmul uses the hardware.

The consequence is direct. If matmul runs efficiently, MFU is high. If matmul stalls on data movement, MFU drops even when the chip is powerful. Raw compute is only useful when the data needed for each operation arrives in time, so the speed at which data reaches the compute units matters as much as the compute units themselves.

Hardware and model design influence one another. A chip can be built to run one model architecture well, and a model architecture can be designed to make the most of a chip. The transformer, for instance, was originally designed by Google to run fast on TPUs and was only later optimized for GPUs.

Hardware choice also depends on memory details. Lower-end and mid-range GPUs may use GDDR (Graphics Double Data Rate) memory, which is one of the hardware details that affects how quickly data can reach the compute units. When comparing options, the memory path is part of the picture, not only the advertised compute number.

Token generation latency is measured with two related metrics. Time between tokens (TBT) is the name used by LinkedIn, and inter-token latency (ITL) is the name used by NVIDIA. Both describe the delay between consecutive generated tokens, which is what a user perceives as the model typing. When comparing reported numbers, check which metric a team means.

Not every delay has the same effect. An experiment by Anyscale showed that 100 input tokens have roughly the same impact on overall latency as a single output token. A long prompt is therefore not free just because it produces little output. This matters for caching later in the article, because the input side of a request carries real cost.

User expectations also shift over time. In the early days of chatbots, some people complained that responses came too fast and seemed unnatural. As people became familiar with chatbots, that stopped being the case, so the latency target a service should aim for depends partly on what its users have come to expect.

Batching for inference starts with a service decision: split the inference API into an online path and a batch path. Online requests are the ones where latency matters most, such as a user waiting on a response. Batch requests are the less urgent ones, which can be sent to a batch API and processed without the same immediacy.

Separating the two lets the service focus its capacity on the online requests first. Routing less urgent work elsewhere protects latency for the requests that users are actively waiting on. The source frames this as the first design decision in batching, before any question of how requests are grouped together.

Caching in inference means saving work that would otherwise be repeated. For transformer models, prefilling populates the initial KV cache, which stores the key and value data the model computes for the tokens it has already processed. Prefilling is the step that fills this cache before generation proceeds, so later steps can reuse that stored work instead of recomputing it from scratch.

Read the diagram from left to right as the sequence of tokens in the prompt. Prefilling creates one stored entry per processed token, so the first four cells are filled before generation begins. A newly processed token, shown as the fifth cell, adds its own entry, which is why the cache grows as the sequence is extended. The point of storing these entries is that the model does not need to recompute them for the tokens it has already handled.

The reason prefilling pays off is that the system prompt of an application is usually known in advance. Developers write it once and it is repeated across many requests. The exact user queries are the part that is hard to predict. So the stable portion of each request can be processed ahead of time and kept in the cache, while only the unpredictable portion must be handled fresh for each request.

Prompt caching applies the same idea at the level of prompts: if a prompt has been processed before, store the result and reuse it rather than recomputing it. Consider a support assistant whose system prompt is identical for every customer. The first request computes the whole prompt. A later request from a different customer reuses the stored system prompt work and computes only the new customer question.

In the second request, the computation for the system prompt is skipped because its stored result is reused. Only the user query, which differs from request to request, is computed. This is why a fixed system prompt makes caching useful: the portion that repeats is exactly the portion that can be stored.

Prompt caching varies by implementation, so the first step is to confirm whether a given system caches whole prompts, prefixes, or only earlier messages in a session. The behavior of one implementation does not automatically transfer to another.

One example from the source is llama.cpp, which has prompt caching, but it appears to cache only whole prompts, and it works for queries within the same chat session. Caching whole prompts means a new prompt that merely shares a prefix with an old one may not benefit. A system that is strict about whole prompts gets no help from a request whose stable opening matches but whose ending differs.

In a long conversation, this kind of cache appears to store the previous messages and process only the newest message. That pattern saves the most work exactly where a conversation keeps growing. Even so, the documentation for such features can be limited, so the behavior is best confirmed in the version you run rather than assumed from a description.

Token generation is bandwidth-heavy. Each generation step requires transferring the entire model's parameters from the accelerator's high-bandwidth memory to its compute units. Because the model produces only one token at a time, that step consumes only a small number of FLOP/s. This is a form of computational inefficiency, and it is the situation speculative decoding tries to address.

The source pack names speculative decoding as the remedy for this inefficiency but does not describe its internal steps, so this article does not claim how it is carried out. What the source does establish is the problem it targets: generating one token at a time moves a large amount of parameter data for a small amount of arithmetic. The limits of speculative decoding in a specific deployment should be checked against that deployment's own measurements.