Evaluating AI Systems Across Usage Contexts

Imagine a dashboard that summarizes how well an AI system is doing. One line on it starts sliding downward over several weeks. The natural first question is whether the whole system got worse. The second question is harder: if it did not get worse everywhere, where did the damage actually happen? A single average cannot answer that by itself. It compresses thousands of individual outputs into one figure, and that compression is exactly what can make a localized problem look like a broad one.

This article follows one real diagnostic situation from the source. A quality signal had moved, and the team needed to decide whether the system had regressed across the board or whether something narrower was responsible. The method that resolved it was not a new model or a rewritten prompt. It was a disciplined habit of going back to individual traces and reading them. The goal is to understand that habit well enough to apply it yourself, and to recognize the trap it helps you avoid.

A regression is a measurable decline in how a system performs compared with an earlier state. When the decline appears in an aggregate number, there are two very different explanations. In the first, every category of input got a little worse. That pattern suggests a systemic cause, such as a change to a shared model or a shared prompt. In the second, one category of input got much worse while the rest held steady. That pattern suggests something specific to that category. The aggregate can look identical in both cases, so it cannot tell you which story you are in.

The useful way to picture this is as a hierarchy. The overall score sits at the top, and beneath it sit the usage contexts that the score was built from. The overall number is a summary of those slices, so a drop in the top line can come from every slice moving a little or from one slice moving a lot. The diagram below shows the structure, using the two kinds of input named in the source case.

The source names the central pattern directly: a metric drift that was a single-specialty problem, masquerading as a systemic one. The word masquerading is the key. The symptom looked as if it came from the whole system, while the cause lived in one narrow slice of the work. Acting on the surface symptom, for example by retuning the entire system, would have spent effort in the wrong place and left the actual problem untouched.

The two patterns look different once you separate the slices. In a single-specialty drift, one slice falls sharply while the others hold steady. In a systemic drift, every slice falls by a comparable amount. Both can produce the same downward line on the dashboard, which is why you cannot decide between them from the total alone.

The figure is illustrative. It shows the shape of the two patterns, not measured values from the source. The practical lesson is that you must look at the slices, not just the total, before deciding what kind of problem you have.

A per-trace loop is the practice of working through the individual traces behind a metric, either one at a time or in a deliberately chosen sample, instead of staring at the aggregate. A trace is one recorded instance of the system handling an input and producing an output. Reading traces individually lets you see what the average was built from. You can notice that certain outputs share features, and then check whether those features line up with the category of input that produced them.

The starting point matters. In the source case, the team did not sample traces at random. They sampled forty traces that an inline judge had flagged as low confidence. Starting from flagged traces focuses attention on the outputs the system's own checks were already uneasy about. That makes the review faster and more informative than reading traces at random, because the flagged set is already enriched for the problem you are investigating.

The loop is a sequence of narrowing steps. Each step asks a more specific question than the one before it, and the answer to the last question tells you where to look next. The source case followed this order: an aggregate drop was flagged, a sample of low-confidence traces was pulled, the traces were read, the findings were sorted by the input category they came from, and those categories were compared against what the system was built to handle.

The step that does the real narrowing is the sort. Reading traces tells you what the outputs look like. Sorting them by the input category they came from tells you where they came from. In the source case, the sort revealed that 33 of the 40 flagged traces came from one orthopedic clinic. That single fact turned a vague system-wide worry into a specific suspect.

The table below is an illustrative set of invented traces, not data from the source. It shows the kind of record a reviewer might produce while working through the loop. Each row is one trace, with its specialty, a note about its shape, and whether the reviewer judged it a match for the style the team expected.

Working through a table like this, a reviewer does not stop at the count of mismatches. The useful question is whether the mismatches share a single cause in the inputs. When they do, the cause can be a specialty, a note template, or a visit type. The next step is then to test that hypothesis against more traces from the same category, rather than concluding from five rows.

The regression in the source came from a mismatch between training data and usage context. The system had been trained on family-practice notes, then used for orthopedic visits, whose notes had a different shape. The model's outputs were still competent in the abstract, but they did not fit the setting where they were being read. Outputs can be technically accurate yet fail the style and shape expected by their readers, and this happens when the usage context differs from the training data.

A useful lesson is that a new usage context can produce a regression even when nothing in the system changed. The orthopedic clinic came online in week four, so the workload shifted under a system that had not been adapted to it. The timing of the drift lined up with a change in workload, not with a change in the system itself.