The Eval-Driven Loop (Chapter 9)
An eval-set-plus-prompt-stack is a small artifact made of a labeled eval set of a hundred examples, a version-controlled documented prompt, and a stack wrapper.
Why a Demo Needs Evidence
Anyone can show a prompt that works on one good input. A client or hiring manager, however, wants to know whether the prompt keeps working across many inputs, and what happens when it does not. Chapter 9 answers this with a small artifact called an eval-set-plus-prompt-stack. It is a compact package that lets you test a specific prompt against known-correct examples, run the same mechanism on live data, and report the results in a form that a working forward-deployed engineer would use.
The source is explicit that this artifact does not have to be large. The goal is the smallest credible piece of work that demonstrates the eval-driven loop. Once you understand the artifact's parts and its results format, you can build one yourself and present it convincingly. This article walks through those parts, traces how they connect, shows how to read a run report, and finishes with a practice task in which you structure your own report.
Three Parts of the Artifact
The artifact has exactly three named parts. Each has a specific job, and the demo is only convincing when all three are present and working together. The source defines the whole artifact as a small package made of a labeled eval set of a hundred examples, a version-controlled documented prompt, and a stack wrapper.
An eval-set-plus-prompt-stack is a small artifact made of a labeled eval set of a hundred examples, a version-controlled documented prompt, and a stack wrapper.
Tracing a Live Customer Input
The source asks you to read the three parts as a chain of dependencies. The eval set supplies labeled examples. The prompt is the version that gets graded against those examples. The stack carries the demo through real-time processing on customer data. The run then produces a result, which is reported. The key point is that the eval set and the prompt meet inside the run, while the stack is the route by which customer data reaches that run.
Follow one live input through the artifact. Suppose a customer sends a new support message while the demo is running. The message does not go to the eval set; it enters through the stack wrapper, which passes it to the versioned prompt. The prompt produces an output. That output is what the customer sees in the demo. Separately, the eval set has already been run against the same prompt, and the scored results from that run are what get written into the report. The live path and the grading path share the prompt but are otherwise distinct.
Read the diagram from left to right. The live customer input enters the stack wrapper first, and the wrapper hands it to the prompt. The output that the customer sees comes from the prompt. The eval set reaches the prompt from a separate direction, and its scored results flow into the run report. Notice that the report is not a product of the live input; it is a product of grading the prompt against known examples.
The Loop From Report to Fix
The artifact is more than a snapshot. The eval-driven loop describes what happens after a run. You run the eval set, read the report, fix the prompt, and run again. The report tells you where the prompt fails. The fix changes the prompt. The next run then shows whether the change helped. Because the prompt is version-controlled, each pass through the loop leaves a traceable record of what was changed and what result followed.
After a run reveals failures, the next change is to the prompt, guided by the fix attached to each failure category. The cycle then returns to the start: the changed prompt is run against the same eval set, so the new report can be compared with the old one. Using the same examples each time is what makes the comparison fair.
Reading a Run Report
The results are presented as a run report. According to the source, the report starts with the overall count, then splits the failures into categories, then gives a fix for each category. A run report gives total runs, passes, and failures, with failures grouped into categories. Each category is paired with a fix, which is what makes the report useful to a client or hiring manager.
Reading a hypothetical run report
A run of the eval set against a prompt produced 100 total runs, 82 passes, and 18 failures. The 18 failures fall into three categories: missing required field (9), wrong category label (6), and empty model output (3). What does each number tell you, and which part of the artifact does each category point to?
- 1Check the overall count first
100 total runs with 82 passes and 18 failures means the prompt passed 82 of the 100 labeled examples. The totals must add up: 82 plus 18 equals 100. A mismatch here would mean the report itself is wrong.
- 2Read the failure categories
The 18 failures are split into 9, 6, and 3, which sum to 18. Each category describes a different kind of miss, so each one points to a different place to look.
- 3Map each category to a part of the artifact
Missing required field suggests the prompt does not ask clearly enough for every piece of output, so the prompt is the first place to look. Wrong category label suggests the prompt does not define the labels precisely. Empty model output may point to how the stack wrapper handles the model's response, so the wrapper is the place to check for that category.
- 4Confirm each category has a fix
The report must pair every category with a fix. For example, the missing-field category might be paired with a fix that tightens the prompt's output instructions, and the empty-output category might be paired with a fix to the wrapper. A category with no fix leaves the next step of the loop undefined.
The report shows that 82 of 100 examples passed, and that the 18 failures are concentrated in three categories. Each category points to a specific part of the artifact, and each has a fix that can be recorded in the next prompt version.
The map above shows the structure the source recommends. Each failure category is a key, its count is part of the value, and the fix is paired with it in the same row. Nothing in the report should float free of a fix. When a client reads it, every number leads directly to an action.
Mistakes When Building the Report
Common mistake Reporting only the pass rate
What it looks likeThe report says the prompt passed 82 percent of the examples and stops there.
Why it failsA single percentage does not say what went wrong or where to look. The source's report structure starts with totals but then splits failures into categories.
Do this insteadGroup the failures into categories and show each category's count.
Common mistake Listing failure categories without fixes
What it looks likeThe report lists three categories with counts but no next step.
Why it failsThe source specifies that each failure category is paired with a fix. Without a fix, the report describes the problem but does not move the loop forward.
Do this insteadAttach one fix to every category, even if the fix is a short description of what to change.
Common mistake Changing the prompt without a version record
What it looks likeThe prompt is edited in place, and the next report cannot be traced to a particular wording.
Why it failsThe source describes the prompt as version-controlled. If changes are not recorded, a score change cannot be attributed to the change that caused it.
Do this insteadSave each prompt revision as a new recorded version and cite that version in the report.
Common mistake Confusing the live path with the grading path: the demo's output on a customer input is not the same thing as the scored run report.
Build Your Own Run Report
Using the structure from this article, draft a run report for a hypothetical prompt tested on a small labeled eval set. Start with the total runs, passes, and failures. Group the failures into at least two categories. For each category, write one fix that points to the prompt or to the stack wrapper, and state which part of the artifact that fix belongs to.
Hints
- Make sure passes and failures add up to total runs before you write anything else.
- A good category describes a specific kind of miss, such as a missing field or a wrong label, rather than a vague problem.
- Ask yourself whether each fix changes the prompt, the stack wrapper, or the eval set, and write that down next to the fix.
Key Takeaways
- An eval-set-plus-prompt-stack has three parts: a labeled eval set of a hundred examples, a version-controlled documented prompt, and a stack wrapper.
- The eval set and the prompt meet inside the run, while the stack wrapper is the path by which real customer data reaches the demo.
- A run report starts with total runs, passes, and failures, groups failures into categories, and pairs each category with a fix.
- The eval-driven loop runs the eval set, reads the report, fixes the prompt, and runs again, so each change can be compared against the same examples.