Latency Metrics for Foundation Models

Picking a foundation model is easy to get wrong if you judge it by one number. Imagine a model whose answers are excellent but which takes so long to respond that users give up before it finishes. Now imagine a model that answers quickly and well but costs so much per request that the product cannot pay for itself. In both cases the model is a poor choice, even though its raw quality looks impressive on paper.

Evaluating a model therefore means keeping three forces in view at the same time: the quality of what it produces, how long users wait for it, and what it costs to run. This article follows how those three forces interact, which latency metrics describe the waiting part of the experience, and how to make a sensible choice among them.

Latency sounds like a single quantity, but for foundation models it is measured in several ways. Each metric captures a different part of the experience a user has while waiting. Read the timeline below from left to right: a request is sent, the first token arrives, further tokens follow one after another, and the response finishes with its final token. Each metric covers a different stretch of that path.

The common metrics are separated by the part of the response they describe. A streaming chat interface cares a great deal about when the first words appear, while a batch report generator may care only about when the entire response is finished. Choosing the right metric depends on what your application is trying to protect. The three metrics below are the ones you will meet most often.

Latency is not a fixed property of a model. It also depends on the prompt that is sent and on the sampling settings used to generate output. Output length is the most direct driver, because each extra output token adds generation work before the response can finish. That gives builders a practical lever: the wait a user observes can be shaped by how the request is written and how generation is controlled.

Optimizing several objectives at once is a field of study known as Pareto optimization. The core technique is to rank your objectives by how firm they are. A firm objective acts as a filter: any candidate that fails it is removed outright. Only after filtering do you compare the surviving candidates on the softer objectives.

This ordering matters. If you compare everything on one blended score, a strong result on one dimension can quietly excuse a failure on a dimension you cannot give up. Ordering the objectives stops that from happening: a model that breaks a firm requirement never reaches the comparison stage, however good its other numbers look.

The source of cost depends on how the model is delivered. The two main routes are calling a model through an API and hosting your own model. The diagram below contrasts what you pay for in each case. The cost of an API is tied to tokens, while the cost of self-hosting is tied to compute.

Scale changes which delivery approach is cheaper. With a model API, the cost per token stays fairly steady as your volume grows, so the total bill rises in direct proportion to usage. With a self-hosted model, the cost per token can drop substantially as volume grows, because the compute spending is largely fixed once a cluster is in place.

Because the economics shift with scale, the choice between an API and self-hosting is not permanent. Companies need to reconsider it as their usage changes. For an API, the scale criterion has an extra dimension: the service must be able to handle your volume at all, not just at a reasonable price.

Turning the three forces into something you can check requires writing criteria down. A criteria table lists each dimension, the metric used to measure it, a benchmark that supplies the test conditions, a hard requirement that a model must meet, and an ideal target that a model should aim for. The table below is an illustrative example for a hypothetical chat application.

Once the criteria are written, selection proceeds in two stages. The first stage asks what the best achievable performance is, meaning how well the strongest candidates can do on the quality measures that matter, and it applies the firm requirements to remove candidates that cannot meet them. The second stage places the remaining candidate models on a map whose axes are cost and performance, and picks the one that delivers the most performance for the money.

The process is more nuanced than two stages suggest, and it does not happen only once. The right model can differ across adaptation techniques. When you use prompt engineering, you might begin with the strongest available model to test whether the task is feasible at all, then work backward toward smaller models to see whether they are good enough. When you finetune, you might begin with a small model so you can test your code cheaply, then move toward the largest model that still fits your hardware constraints, such as a single GPU.