Defining Success Metrics for AI Systems

Suppose you have built a system that reads incoming documents and flags the parts that need attention. Before anyone relies on it, you need an honest answer to a simple question: does it actually do the job? Answering that question requires a set of test cases, and the quality of your answer depends almost entirely on which test cases you chose. This article is about choosing them well.

The working example throughout is a contract-review system. The reasoning, however, applies to any system whose output must be checked against real work.

The source describes a contract-review deployment built for a mid-market commercial real estate firm. The system was meant to read inbound vendor contracts, identify clauses that departed from the customer's standard playbook, and produce a redline memo. The practical purpose was that a general counsel who used to spend about an hour on a review should be able to finish it in ten minutes.

This purpose sets the requirement for testing. The system is only useful if it performs well on the documents the firm actually handles, so the test cases have to resemble those documents. A test set built from documents that are easier, stranger, or simply different from the firm's real work would not tell the firm whether the system is ready.

The evaluation set in the source was assembled quickly, in the first three days of the validation work. The deputy general counsel pulled forty-two real contracts from the six months before the project began.

The forty-two contracts were not all alike. They fell into three groups, and each group carries a different level of difficulty. Because of this, a single overall score is not the whole story. Separating the groups lets you ask how the system behaves on simple work and on demanding work, rather than hiding both behind one average.

The groups together account for the whole set of forty-two. The simple group tests whether the system leaves clean documents alone, the middle group tests routine redlining, and the complex group tests whether the system can handle heavy, high-stakes revisions. Because the groups are reported separately, a weak result on the complex contracts cannot be hidden by strong results on the twelve NDAs.

The word representative carries most of the weight in the definition of an evaluation set. A representative evaluation set is built from real-world examples that reflect the mix of work a system will actually face. A set that is large but made up only of easy or unusual cases will report misleading performance.

The aim is a set whose mix of cases matches the mix the system will meet in practice. In the source, this was achieved by having the person who would be affected by the system's output select the documents, rather than having an outside party guess at what typical work looks like. Evaluation sets should be chosen with input from the people who do the real work.

Three ideas in this part of the course are easy to confuse because they all concern measurement. The evaluation set is the material being tested. A success metric is a quantifiable measure of how well the system is doing its job. A validation threshold is the specific target the metric must reach before the system is considered acceptable.

In the contract-review case, the customer's success metric was the number of attorney review minutes spent per contract. The validation threshold attached a target to that metric: a sixty percent reduction in those minutes, with no missed redlines on the cases the firm cared most about. The threshold therefore combines a measured reduction with a second condition about accuracy.

The metric tells you what to measure. The threshold tells you what result counts as success. A reader who knows only the metric cannot tell whether the system passed. A reader who knows the threshold knows exactly what passing means.

Now trace how a single evaluation example moves through the process. Each contract from the set is passed through the system. The system produces a redline memo. The review effort is measured as the metric. That measured value is then checked against the threshold, which yields a pass or fail decision for the system.

Notice that the outcome depends on both conditions. The time reduction alone is not enough. The result for the system is the combined judgement of the metric and the threshold, applied across the whole evaluation set.