Contract Analysis Script

Before a team trusts an automated contract review tool, someone has to watch it work on real material and judge the result. This article follows one such moment: a demo session in which a contract analysis script was run in front of a deputy general counsel (deputy GC). The session is useful as a case study because it shows the whole loop, from a live contract going in, to a memo coming out, to human feedback being turned into something the team can keep and reuse.

The setting was simple. The session took place on a Friday afternoon in the deputy GC's office. She used her own laptop, and the keyboard stayed in her hands the whole time. The contracts were not sample documents. They were five real inbound contracts that had been pulled from her email that same morning. Working with live material is what separates a demo from a test: the questions being asked are the ones the business actually cares about.

The core interaction is a single action repeated for each contract. A contract is dragged into the script. The script then runs, which took fourteen seconds on the contract in this session, and it produces a memo. The memo is the artifact the reviewer reads. The script's job is to turn a raw contract into a readable document that a lawyer can check against her own judgment.

The diagram shows the order of operations. The script sits between the contract and the reader, and the reader's response is a separate step that happens after the memo exists. The script does not decide what the reviewer concludes. It supplies the material for that conclusion.

Once the memo was open, the deputy GC read it and gave a short verbal response on each of the five contracts. For most contracts the answer was simply yes. For two, she named a specific problem. The way she spoke about each contract is a model of how feedback on a tool should be expressed: a clear verdict, and a specific location or reason when the verdict is negative.

Timing also matters. Of the five contracts, two had issues, and both were flagged within ninety seconds. That speed is less a benchmark than a sign of how the review was structured. Flagging happened immediately after the reader engaged with the memo, while the contract and its memo were still fresh. The ninety seconds covers the reading and the decision, not the script's fourteen-second run on its own.

It helps to think of each contract as moving through a small set of states. It starts with no known issue, moves to an issue being flagged when the reader identifies one, and can then move on to becoming a stored eval case. The data flow below follows the indemnity issue through that progression.

The label on case #44 is worth dwelling on. Here the answer came from a human reviewer, the deputy GC, rather than from the script. The corrected risk grade became the label. That is why the session is useful as a source of evaluation material: the human judgment is written down in a form that can be reused, instead of living only in the memory of the meeting.

The two flagged contracts were not the same kind of problem. Seeing them side by side makes the difference clearer than describing either one alone. The comparison below sets out what each issue was and what feedback it produced.

The two columns share a structure. Each issue was flagged within the same review window and each was added as an eval case. What differs is the kind of feedback that made the issue useful. Read the subsections below for the details of each.

A five-contract demo supports some conclusions and not others. It can show that the script produces a memo for each contract in a live setting, and that the reader was able to flag specific issues in it. It also shows the kinds of issues that appeared, namely a missed clause and a misgraded risk, and gives a concrete reusable example for each through the eval cases.

It cannot show the script's overall performance. Five contracts is too few to estimate how often the script succeeds or fails across the range of inbound material the business receives. A team that wants that answer would need a larger evaluation, drawn from a broader set of contracts and judged against labels that are consistently applied. The demo is best read as a source of specific feedback, which is valuable precisely because it is concrete.