Sampling Production Traces

Once an AI product is running with real users, the most useful information about its behavior is the record of what actually happened in each interaction. Those records are called production traces. A forward-deployed engineer, or FDE, does not treat these records as a one-time report. Instead, the FDE works through them on a repeating daily cycle. This article calls that cycle the per-trace loop. Each pass through the loop takes a small slice of real behavior, turns it into a candidate improvement, and checks whether the improvement really helps before it reaches users.

It helps to think of the loop as a sequence that starts with new material arriving and ends with a decision. The source describes the loop as the place where the FDE spends the day, so the stages below are the ones that repeat from one day to the next. The four stages are sampling production traces, identifying a pattern, coding a fix, and testing the change.

Notice that the loop only moves forward through the fix when there is evidence from real traces. A pattern is the bridge between what the FDE read and what the FDE changes. Without that bridge, a change would be a guess. The final check, running the eval set, is what separates a plausible idea from a change that is actually known to help.

The source describes two ways sampling can happen. Sometimes it is random, which gives a broad, unbiased view of what the product is doing. Often, though, the sample is weighted toward two kinds of cases. The first is traces the inline judge flagged. The inline judge is a signal already attached to the product's behavior that points at traces that may have gone wrong. The second is traces the customer manually marked, which point at interactions a person using the product found worth noticing.

Random sampling is the choice to let the review set reflect the pile as a whole. Its strength is breadth: it shows the common behavior of the product without favoring any particular kind of case. Weighted sampling is the choice to tilt the review set toward cases that already carry a signal, namely traces the inline judge flagged or the customer marked. Its strength is focus: it puts more of the FDE's limited reading time on interactions most likely to reveal a problem. The source does not say the two methods are used in a fixed proportion, so the FDE decides the balance for the situation.

A pattern is a repeated shape across several sampled traces. Once the FDE sees that shape, the next question is what change might address it. The source lists four families of changes a pattern might point toward. A prompt change alters the instructions the model receives. A tool-description change alters how the model understands the tools it can call. A context-window strategy change alters how information is packed into what the model sees at once. A model swap replaces the model itself. Each family changes a different part of the system.

The choice among these families is driven by the pattern itself. The source does not give a rule for matching patterns to families, so this matching is a judgment the FDE makes from the traces in front of them. What the source makes clear is that the chosen fix is then coded, which turns it into a concrete change that can be tested.

A fix that looks right on a handful of traces may still make the product worse elsewhere. To check the change more broadly, the FDE runs the eval set from Chapter 9 against the changed system. The eval set is the instrument for this test. Its role is to produce a score, and the score is what the loop uses to judge the change. The test happens before the change is shipped, which means the eval set stands between a coded fix and real users.

The loop ends with a binary decision that depends on the score. If the change improves the score, it ships. If it does not, it goes back to the bench for further work. Because the decision is tied to the eval set's score rather than to how convincing the change looked when it was written, each shipped change rests on a measured improvement.