Labeling Cases for Evaluation

Suppose a firm builds a system that reads contracts and flags the clauses a lawyer should look at. Before anyone can trust it, someone has to decide whether it is doing a good job. A common approach is to gather a collection of contracts where the right answer is already known, run the system over them, and compare its output against those known answers. That collection is called an evaluation set.

Most people picture the evaluation set as a fixed test bench: you build it once, then measure the system against it. The source engagement shows something different. The set changed while the work was underway, and that change turned out to matter as much as the measurements themselves. In this article we follow one evaluation set through an engagement and ask what happens to the set itself over time.

The clearest way to understand the evaluation set is to watch how its size changes. The set started the engagement with forty-two cases and finished with one hundred and thirty-one. It did not begin at its final size and stay put. Each stage of the work added cases, so the set at any point in time was a snapshot of what the team had labeled so far.

The diagram shows the set moving from forty-two cases to one hundred and thirty-one. The source does not give the number of cases at month three, so the diagram does not supply one. What it does show is that the growth happened in stages, and that one stage, month three, is where the set began to play a role nobody had planned for.

The set grew for two main reasons. Each reason signals something different about what the set still lacks, so it helps to keep them separate in your mind.

A detail in the source is easy to skim past. Each surprising case was added to the set the same day, with the deputy general counsel's label attached. The label is what turns a contract into a case. A contract on its own is just a document. A contract with a reviewer's answer attached is a test the system can be checked against later.

Adding the case quickly, while the surprise was fresh, meant the reasoning behind the disagreement was captured rather than forgotten. The timing is part of the rule. A label recorded weeks later may not carry the same reasoning, which is why the source ties the addition to the same day.

The same growth reasoning can be applied to a new incoming contract. The flowchart below follows the rule the source describes: first check whether a reviewer has supplied a label, then check whether one of the two growth triggers applies. The source describes these two triggers as the main sources of growth; it does not describe any other admission path, so the chart does not add one.

A system's output is judged by comparing it against the known answers in the set. The diagram below shows that comparison: a contract with a known answer goes to the system, the system's output is set against the reviewer's answer, and the result is a quality judgment.

By month three the evaluation set had become the firm's internal reference for what good meant in contract review. The source describes this as a result that had not been planned for, and it became the most valuable artifact of the engagement.

The reason this matters is that a set of labeled cases encodes judgments. Each case carries a reviewer's answer about what the correct output is. When enough of these judgments accumulate, and they reflect the real shapes of incoming work, they begin to answer the question of what good looks like. People can point to a case and say that is the standard. The set stops being only a measuring tool for the system and becomes a shared record of the firm's expectations.