Improving AI Products through Iterative Development
The per-trace loop starts by reading individual sampled traces with their actual input and actual output, rather than the dashboard or aggregate numbers.
Two Teams, Same Dashboard
Imagine two teams that ship the same AI assistant. Both watch a dashboard full of averages. Both see the numbers drift slightly downward in the third week after launch. The first team responds by tweaking settings and hoping the averages recover. The second team opens a handful of individual conversations, reads what the user actually typed and what the assistant actually answered, and discovers that one kind of request is being handled badly again and again. The second team has begun working through the per-trace loop.
What do you think happens?
Which team is more likely to find the real cause of the downward drift?
Reveal answer
The team that opens individual conversations and reads the exact input and output
The source describes the second team as the one that has begun working through the per-trace loop, because it reads the actual wording of individual exchanges where the failure lives.
This article explains how that work is organized, and why it is the process that turns a fragile launch into a product people keep using.
What a Sampled Trace Contains
A trace is the record of one exchange between a user and an AI product. A sampled trace is one of these records picked out of the stream of real activity rather than invented.
The discipline asks for something that feels slow. You read the input the user supplied and the output the product produced, in their exact wording. Summary numbers compress many exchanges into one figure, and that compression hides the specific words where a failure lives. Reading the raw exchange keeps those words visible.
Forming a Working Hypothesis
A single trace rarely tells you enough by itself. The loop asks you to read enough traces in sequence to form a working hypothesis about what is going wrong. The hypothesis is therefore a pattern you notice across several exchanges, not a conclusion drawn from one unusual reply.
From Several Traces to One Hypothesis
You have read five sampled traces from an AI assistant in the same week. Each trace contains a request to reschedule an appointment, and in each one the assistant replies with a general explanation of scheduling instead of an action. What working hypothesis should you form?
- 1Read the traces in sequence
Look at each exchange's exact input and exact output. A single trace with this pattern might be dismissed as an odd reply.
- 2Look for the shared element
The same kind of request appears in every trace, and the same kind of unhelpful response follows it.
- 3State the pattern as a hypothesis
The working hypothesis is that rescheduling requests are being answered with general explanations rather than the expected action. It is a claim about a pattern across several exchanges.
- 4Hold it open to checking
The hypothesis is useful only if it can be tested later, which sets up the next stage of the loop.
The working hypothesis is a pattern across several traces: rescheduling requests are being handled with general explanations instead of action.
The Stages of the Per-Trace Loop
It helps to see the loop as a sequence in which each stage produces something the next stage uses. Reading sampled traces produces a hypothesis. The hypothesis is encoded as an eval case. The eval case is added to the standing eval suite. The suite is then used to grade a fix. Notice that the fix is not the end of the chain: its grade comes from the suite, which already holds everything the team has learned so far.
Encoding a Hypothesis as an Eval Case
A hypothesis is useful only if it can be checked again later. The loop therefore encodes it as an eval case: a test built from the hypothesis that captures the kind of input that exposed the problem and the behavior a good answer should show. Writing the case is what converts an observation you made while reading into something the team can run repeatedly.
The eval case then becomes part of the standing eval suite. The word standing matters. The suite is not rebuilt for each change; it accumulates. When a fix is proposed, it is graded against the suite, so the fix must handle the new case and must also keep passing the cases the team wrote earlier.
Grading a Proposed Fix Against the Suite
When a team proposes a fix, the suite decides whether the fix is accepted. The fix must handle the new case that came from the latest hypothesis, and it must keep passing the cases written in earlier cycles. A fix that repairs the new case but breaks an older one has not passed the grade.
A proposed fix passes the newly added eval case but fails two cases written in earlier weeks. Using the loop as described, what should happen to the fix, and what should the team do next?
Hints
- The source says the fix must also keep passing the cases the team wrote earlier.
- Consider what the suite is for: it accumulates and is not rebuilt for each change.
Why Repeating the Loop Compounds
The source describes the loop's shape as unglamorous, and it is also what produces the week-over-week improvement curves that separate a deployment customers keep using from one they quietly turn off about six weeks after launch. Those curves are not a single event. Each cycle of reading traces, forming a hypothesis, encoding it, and grading a fix adds to what the team knows and what the suite checks.
Because every cycle leaves an eval case behind, the suite grows alongside the product. Later fixes are graded against everything already learned. That is why the loop is best understood as the mechanism behind sustained improvement rather than as a one-off review. In the source's words, the loop is the work.
Mistakes Beginners Make with the Loop
Common mistake Starting from the dashboard averages
What it looks likeAdjusting settings because the weekly average dipped slightly.
Why it failsAggregate numbers compress many exchanges and hide the specific words where a failure lives.
Do this insteadOpen sampled traces and read the exact input and exact output first.
Common mistake Forming a hypothesis from one unusual reply
What it looks likeDeclaring a systematic problem after a single strange answer.
Why it failsA single trace rarely tells you enough by itself.
Do this insteadRead several traces in sequence and look for a pattern across them.
Common mistake Leaving the hypothesis as a note
What it looks likeWriting down an observation that is never turned into a test.
Why it failsA hypothesis is useful only if it can be checked again later.
Do this insteadEncode it as an eval case and add it to the standing suite.
Common mistake Rebuilding the suite for each change
What it looks likeTesting a fix only against the newest case.
Why it failsThe suite accumulates, and a fix must also keep passing earlier cases.
Do this insteadGrade every proposed fix against the full standing suite.
Putting the Loop to Work
Your product's weekly averages drift down after launch. Describe the first three actions you would take under the per-trace loop, in order, and explain why each one comes before the next.
Hints
- Start with what the loop says to read before forming any conclusion.
- Think about what must be written down so that a later fix can be checked.
Key Takeaways
- The per-trace loop begins by reading individual sampled traces with their exact input and output, because aggregate numbers hide the specific wording where a failure lives.
- A working hypothesis is a pattern noticed across several traces read in sequence, not a conclusion drawn from one unusual reply.
- A hypothesis is encoded as an eval case and joins the standing eval suite, which accumulates rather than being rebuilt for each change.
- A proposed fix is graded against the full suite, and repeating the loop adds to what the team knows, producing week-over-week improvement.