Measuring Model Performance

Imagine a contract-review system that has just finished its first full run and produced an overall score. That number tells you how the system did on average, but it does not tell you where it went wrong. Two different systems can earn the same aggregate score while failing in completely different ways. One might consistently miss a single high-stakes clause type, while the other scatters small errors across many documents. Both produce the same headline number, yet the right next step is different for each.

This article follows one idea from the source material: the diagnostic value of a first run is often higher than any aggregate score it produces. Before you tune anything, you need to know which situation you are in. The article covers why measurement is set up at the start of a project, how a gap between results and expectations is judged, how individual failures are read, and how to tell whether a stalled metric is a prompt problem or a rubric problem.

A common instinct is to scope the project, then move straight into building. Production code gets written, integrations get stood up, and the model gets deployed, and only then does anyone check how well the result performs. The assumption behind this sequence is that measurement can wait until there is something substantial to measure.

The source rejects this sequence directly. During the first week, the team does not write production code, stand up the integration, or deploy the model. Instead, that week is spent building the evaluation set and running baseline performance against it. The reason is that the evaluation set is the instrument that makes the rest of the project measurable. Once it exists, the team can see how the system performs before any optimization has been applied. That starting measurement gives the team and the customer something concrete to discuss, rather than an unspoken expectation that each side may interpret differently.

Trace the state of a project through this comparison. Start with the baseline score and the customer's expected level. If the baseline already meets the threshold, the gap is effectively closed. If it falls short, the size of the shortfall becomes the central fact for planning. The source describes two outcomes. When the gap is small, the engagement is expected to be short. When the gap is large, the source does not promise a fixed schedule; instead, the team and customer can use the shared understanding to adjust scope and timeline together. The value lies in making the gap visible early, while changes are still possible.

An aggregate score averages over every document, so a specific failure can disappear inside it. An aggregate score summarizes results, but the individual failures from a diagnostic run often reveal the specific cause that needs fixing. Pulling one miss out of the score and reading it closely is how you learn what the average hides.

The contrast between the two systems in the introduction makes the point clear. One system that consistently misses a single high-stakes clause type and another that scatters small errors can share an identical score. Only the individual failures reveal which situation you are in, and therefore which change is worth making.

Diagnostic analysis reads misses together to find a shared pattern, then turns that pattern into a testable hypothesis. The next step is a targeted change aimed directly at that pattern. The source describes three prompt changes working together: explicit instructions to scan appendices and definitions, and a structured pre-pass that listed every section of the contract before the redline pass began. The effect was that the C1 gap closed on the next run. The source does not elaborate on what C1 measured beyond this gap, so the lesson here is about the method, not the specific metric.

The first run does not end the process. It supplies the evidence that shapes the next change. The cycle repeats: diagnose the misses, change the prompt or rubric, and re-run against the same evaluation set to confirm or refute the change. Each pass carries a specific piece of knowledge forward, from raw misses, to a shared cause, to a targeted adjustment, and finally to a re-run.

The loop is read as a cycle rather than a single pass. Each re-run either confirms the hypothesis, as when the C1 gap closed, or refutes it, in which case the misses are diagnosed again with the new evidence in hand.

When a metric stalls, the instinct is to change the model or its prompt. The source warns that this is not always right. Before concluding the model is at fault, check whether the rubric is unclear. A rubric is the set of grading criteria that decide whether an output counts as correct. If those criteria are ambiguous, the metric cannot measure the intended behavior reliably, and changing the model would not fix a measurement that was ambiguous to begin with.

The source reports that its C2 gap took longer to resolve and turned out to be a rubric-clarity problem more than a model problem. The fix that worked was a second iteration of the rubric, with worked examples for each risk level. The source says this revision unblocked the metric. The practical lesson is that the rubric is part of the evaluation system. An unclear rubric can make a correct output look wrong, or an incorrect output look right.

Read the flowchart as a sequence of checks. The rubric question comes first because a stalled metric may be a measurement problem rather than a model problem. Only when the rubric is clear and the misses share a pattern does a targeted prompt change follow. Either branch ends in a re-run against the same evaluation set, so the effect of the change can be confirmed.

This scenario is generated for practice and is not drawn from the source. A contract-review system's first run on an evaluation set shows that its misses cluster on limitation-of-liability clauses. Three of the four misses are clauses placed inside an exhibit, not the main body. Work through one pass of the loop: state the shared cause you would hypothesize, the targeted change you would make, and what the re-run should show if your hypothesis is right.