Designing Metrics for AI Systems

Imagine you have built an application that sends user requests to a foundation model and returns the model's answers. In your tests it works well. Then, a few weeks after launch, users start reporting responses that seem wrong, slow, or unsafe. The natural first instinct is to start fixing the problem. But the more important first question is different: how would you have known about it at all? Could you see that something had gone wrong, find where it went wrong, and tell the difference between a single odd answer and a real pattern?

The idea that answers this question is called observability. It asks you to think about failure before failure happens. The rest of this article works through that question, and it is useful to keep it in mind as you read.

The definition has three parts, and each depends on the one before it. First, you need to understand the ways the system can fail. Second, you decide what to measure and what should trigger an alert, based on those failures. Third, the system itself has to be built so that the measurements are possible and a failure leaves a trail someone can follow.

This is why observability is best treated as a property you design into the system, not something you bolt on after an incident. A metric is useful only if the system actually produces the information it measures, and an alert is useful only if it is tied to a metric that matters.

To see why traceability matters, follow one request through an AI application. Suppose the request passes through a retrieval step that looks up supporting material, then a foundation model call, then a post-processing step before the answer is returned. The diagram below shows that path. When a user reports a bad answer, the question is which stage produced the problem.

Notice that the failure is only visible at the end of the path. If the system has recorded what each stage received and produced, you can compare the records and see where the answer first went wrong. If it has recorded nothing except the final answer, you know that something is wrong but not where it started. The trail through the stages is what turns a complaint into something you can act on.

The word detectable carries real weight in the definition. A failure that happens silently, with no signal anywhere, cannot be acted on. Designing for observability means making sure that the failures you care about leave a signal that someone or something will notice in time. A wrong answer that looks like any other successful response leaves no signal unless you deliberately created one.

Traceability is the companion idea. A failure should leave a trail that a person can follow from the symptom back to the request that produced it, and through each step the request took. Records made at each stage, tied together by the same request, are what make that possible. Without them, the system may be running correctly from the infrastructure's point of view while users receive answers that are wrong.

Separating an AI application into components keeps it easier to change and maintain. Each component has a clear job, and you can reason about it on its own. However, the boundaries between components are rarely as clean as a diagram suggests. Several components can take on overlapping responsibilities, and the same job can often be placed in more than one spot.

A guardrail shows this clearly. A guardrail is a check that screens what goes into or comes out of a model, for example to block content you do not want the application to produce. The job of the guardrail stays the same wherever it lives. It can be implemented in the inference service that runs the model, in the model gateway that routes requests, or as a standalone component of its own. What changes is where the check lives and which other parts of the system must know about it.

Placement matters for observability. If the check is buried inside the inference service, its decisions may be visible only in that service's records, so someone investigating a problem has to know where to look. If it is a standalone component, it can report its own measurements separately, which is helpful, but you now have one more piece to monitor and one more place where things can go wrong. In every case, the addition makes the system more capable, safer, or faster, and it also makes the system more complex and creates new ways to fail.

Much of what you already know about monitoring software applies here too. Software engineering has long offered practices for logging, tracing requests, watching resource use, and paging someone when a service degrades. Traditional machine learning has its own practices for tracking model behavior over time. Many of the tools built for these fields carry over to AI engineering, and you should use them rather than reinvent them.

The difficulty is that foundation models introduce failure modes that those existing practices were not built to catch. A failure mode is a particular way a system can go wrong. Because the model's behavior is produced by the model rather than written line by line by your team, the ways it can misbehave do not always match the failures your existing dashboards were designed to show.

The practical result is that the metrics you already collect may be necessary but not sufficient. A service can have healthy resource use and fast responses while still returning answers that are wrong, unhelpful, or unsafe. You need additional metrics and additional design decisions aimed at the failures a foundation model can cause.

Once you have a list of the failures your application can produce, each one suggests what to measure and what should trigger an alert. The map below is an illustrative example of that planning step. The failures listed are plausible ones for an AI application, and each is paired with a signal that could reveal it. The right alert threshold for each depends on your application and is a decision your team makes, so none is shown here.