Contract Issue Identification

Before a team trusts an automated contract review tool, someone has to watch it work on real material and judge the result. This article follows one such moment: a demo session in which a contract analysis script was run in front of a deputy general counsel (deputy GC). The session is a useful case study because it shows the whole loop, from a live contract going in, to a memo coming out, to human feedback being turned into something the team can keep and reuse.

The setting was simple. The session took place on a Friday afternoon in the deputy GC's office. She used her own laptop, and the keyboard stayed in her hands the whole time. The contracts were not sample documents. They were five real inbound contracts that had been pulled from her email that same morning. Working with live material is what separates a demo from a test: the questions being asked are the ones the business actually cares about.

The core interaction is a single action repeated for each contract. A contract is dragged into the script. The script then runs, which took fourteen seconds on the contract in this session, and it produces a memo. The memo is the artifact the reviewer reads. The script's job, in other words, is to turn a raw contract into a readable document that a lawyer can check against her own judgment.

Notice the order. The reader's response is a separate step that happens after the memo exists, not while the script is running. The script does not decide what the reviewer concludes; it supplies the material for that conclusion. Once the memo was open, the deputy GC read it and gave a short verbal response on each of the five contracts. For most contracts the answer was simply yes. For two, she named a specific problem.

It helps to think of each contract as moving through a small set of states. It starts with no known issue, moves to an issue being flagged when the reader identifies one, and then can move on to becoming a stored eval case. A label is the answer the case is checked against. In this session, both flagged contracts were added as eval cases before the demo ended.

The label on case #44 is worth dwelling on. Here the answer came from a human reviewer, the deputy GC, rather than from the script. That is why the session is useful as a source of evaluation material: the human judgment is written down in a form that can be reused, instead of living only in the memory of the meeting. The corrected risk grade served as the label for the indemnity case.

The two flagged contracts were not the same kind of problem. Seeing them side by side makes the difference clearer than describing either one alone. One problem was a missed clause: the script did not surface an auto-renewal clause in section 12 of one contract. The other was a judgment problem: the script assigned a risk grade on an indemnity that the reader judged to be wrong. The first is a gap in what the memo contains; the second is a disagreement about what the memo says.

Of the five contracts, two had issues, and both were flagged within ninety seconds. The speed matters less as a benchmark than as a sign of how the review was structured: the flagging happened immediately after the reader engaged with the memo, while the contract and its memo were still fresh. Ninety seconds covers the reading and the decision, not the script's fourteen-second run on its own.

Five contracts is a small sample, so the demo is best read as a source of specific, reusable feedback rather than a complete performance measure. It can show that the script produced readable memos on real inbound material, that a reviewer could find two concrete problems, and that those problems could be turned into stored cases. It cannot show how often the script misses issues across a full contract portfolio, nor how it performs on contract types that did not appear in this session. Answering those questions would require a larger or different evaluation.