Designing an Evaluation Set for Contract Review

Suppose a firm builds a system that reads contracts and flags the clauses a lawyer should look at. Someone has to decide whether the system is doing a good job. A common approach is to gather a collection of contracts where the right answer is already known, run the system over them, and compare its output against those known answers. That collection is called an evaluation set. Most discussions of evaluation sets treat them as a fixed starting point. This article follows one evaluation set through a real engagement and asks a question that is easy to overlook: what happens to the set itself while the work is underway?

The clearest way to understand the set is to watch how it changes over time. Start with the size. Over the course of the engagement the set grew from 42 cases to 131. That growth did not happen at a single moment. It happened in steps, as new cases arrived and as surprising outputs were recorded. The timeline below traces the main points where the set changed and the state it reached at each stage.

Read the timeline as a record of the set's life rather than a fixed table. The starting collection of 42 cases is the only point where the set had a known size at the beginning. Every later stage added cases, and the set's role also changed: by month three it had become something the team used to judge quality, which nobody had originally planned for.

The growth had two main sources, and each one signals something different about the set. The first source was the arrival of new contract shapes. A contract that looked structurally unlike anything already in the set was a sign that the set did not yet cover that kind of document. The deputy GC added such cases as they came in, so the set kept pace with the variety of real work the firm received.

The second source was disagreement. Every case that surprised the deputy GC, meaning every contract that produced an output she disagreed with, was added to the set. This is a different kind of signal. A new shape tells you the set is missing a type of input. A surprising output tells you that the system and the reviewer diverged on a case that may already resemble others, and the disagreement itself is worth preserving as a test.

Put in one sentence, the growth rule in the source is this: a case enters the set when it reveals a missing kind of input, or when the system's output disagrees with a reviewer's judgment. The diagram below turns that rule into a sequence of checks, so you can follow what happens to a candidate contract at each step.

The flowchart is a generated reading of the two growth sources in the source, not a procedure the source spells out step by step. It asks the two questions the source describes: does the contract have a shape the set does not yet cover, and does the system's output disagree with the reviewer's judgment? A candidate that passes either check is admitted. The terminal box for a candidate that passes neither check reflects the inference that those two sources were the only stated reasons for adding cases.

A detail in the source is easy to skim past. Each surprising case was added to the set the same day, with the deputy GC's label on it. Adding the case quickly, while the surprise was fresh, meant the reasoning behind the disagreement was captured rather than forgotten. A label written weeks later would have to reconstruct why the output looked wrong; a label written on the day records the reviewer's reasoning while it is still clear.

The before and after view shows what a single admission changes. The count goes up by one, and when the case was admitted because of a new shape, the set's coverage of contract types widens. The known correct answers also change: the set now includes one more contract paired with a reviewer's answer that the system can be checked against. The source does not describe the exact content of each intermediate state, so the figure shows only the two changes the source does describe.

By month three, the evaluation set had taken on a role nobody had designed for it. It had become the firm's internal reference for what good meant in contract review. The source describes this as a result the team had not planned for, and it became the most valuable artifact of the engagement.

The reason this matters is that a set of labeled cases encodes judgments. Each case carries a reviewer's answer about what the correct output is. When enough such judgments accumulate, and when they reflect the real shapes of incoming work, they start to answer the question of what good looks like. People can point to a case and say that is the standard. The set stops being only a measuring tool for the system and becomes a shared record of the firm's expectations.