Evaluating Contract-Review Systems

Imagine you have built a system that reads incoming documents and flags the parts that need attention. Before anyone relies on it, you need an honest answer to a simple question: does it actually do the job? Answering that question requires a set of test cases, and the quality of your answer depends almost entirely on which test cases you chose. This article is about choosing them well.

The working example is a contract-review system, but the reasoning applies to any system whose output must be checked against real work. As you read, keep one question in mind: if you were the person responsible for accepting the system, what would you need to see in a test set before you trusted the results?

The contract-review deployment in the source was built for a mid-market commercial real estate firm. The system was meant to read inbound vendor contracts, identify clauses that departed from the customer's standard playbook, and produce a redline memo. The practical purpose was that a general counsel who used to spend about an hour on a review should be able to finish it in ten minutes.

This goal explains why the test cases matter so much. The system is only useful if it performs well on the documents the firm actually handles, so the test cases have to resemble those documents. A test set built from some other kind of document would tell the firm very little about its own work.

The word representative carries most of the weight in that definition. A set that is large but made up only of easy or unusual cases will report misleading performance. The aim is not simply to have many test cases. The aim is a set whose mix of cases matches the mix the system will meet in practice.

In the source, this was achieved by having the person who would be affected by the system's output select the documents, rather than having an outside party guess at what typical work looks like. An invented example reflects the imagination of whoever wrote it. A real example reflects the actual traffic, including the messy cases that a guesser would be unlikely to think of.

The evaluation set in the source was assembled quickly, in the first three days of the validation work. The deputy general counsel pulled forty-two real contracts from the six months before the project began. The contracts were not all alike, and the set was divided into three groups. Each group carries a different level of difficulty, so a single overall score is not the whole story.

Read the three groups together and the set's design becomes clear. Twelve straightforward NDAs had been signed without any changes, so they test whether the system can recognize unremarkable work without flagging problems that do not exist. Twenty-three vendor service agreements needed redlines on between two and seven clauses, which is the bulk of ordinary work. Seven complex landlord-tenant agreements each had redlines on more than ten clauses, and at least one issue in each had required outside counsel.

Three ideas appear in this part of the course, and they are easy to confuse because they all concern measurement. The evaluation set is the material being tested. A success metric is a quantifiable measure of how well the system is doing its job. A validation threshold is the specific target the metric must reach before the system is considered acceptable.

In the contract-review case, the customer's success metric was the number of attorney review minutes spent per contract. The validation threshold attached a target to that metric: a sixty percent reduction in those minutes, with no missed redlines on the cases the firm cared most about. The threshold therefore combines a measured reduction with a second condition about accuracy.

Once the set and the threshold are fixed, judging the system is a sequence of steps. The flowchart below follows that sequence from running the system on the evaluation set through to the pass or fail decision. Trace the path from the top and notice that the decision depends on both conditions, not only on the time measurement.

The same steps apply to the full set. The metric is computed for each of the forty-two contracts and the threshold is checked against the results, with the composition reported alongside. Because the three groups differ in difficulty, the results should be read group by group as well as in total.

The reasoning carries over to other systems. Imagine a system that sorts customer support emails into categories such as billing, technical fault, and account access. If you assembled a test set made only of short, clearly worded emails, the system could score well and still fail on the long, rambling messages that real customers often send.