Evaluating Contract Review Output with Evals

The project is organized into a README, a prompts folder, a src folder, an evals folder, and a traces folder, and each part has a separate purpose.

Programming

Opening the Project Before Reading the Code

Imagine you have a small, working contract review tool. You give it a contract, it sends the contract to a model, and it returns a memo. Now you open the project folder for the first time. Where is the work actually done? Before reading any description of the design, it helps to look at the project as a set of folders and files, because the layout tells you where each responsibility lives.

Keep one idea in mind as you read: a well-organized project lets you find the part responsible for a behavior without reading everything. The rest of this article uses that idea to map the project, trace one contract through it, and then locate the right file when a memo goes wrong.

EASY

Before you read the next section, make a guess. The src folder holds three Python files: main.py, parse.py, and format.py. One of them has to read text out of PDF or Word documents and find the sections and appendices inside them, which is a fiddly job. Another has to be short, since it only receives a contract, calls models, and returns a memo. Which file do you think does which job? Write down your guess now.

Hints

Think about which job is the messiest: reading documents or formatting the final memo.
Think about the job that only receives a contract, calls a model, and returns a memo.

Your attempt (not saved)

The Five Top-Level Parts

The whole project sits in one top-level folder, named contract-review-v1. Inside it are a README and several subfolders. Each subfolder groups one kind of material, so the folder name tells you what kind of thing to expect inside it.

Which folders and files does the project contain, and how do they nest inside one another?

The README is a single page that explains how to run the tool and what it does. The prompts folder holds the instructions given to the model. The src folder holds the Python code that runs the tool. The evals folder holds the evaluation suite. The traces folder holds per-run JSON dumps, which the source says are used for the per-trace loop. The source describes these last two folders only briefly, so this article does not explain how the evaluation suite reads the traces. What matters for locating a bug is that the behavior lives in src and the instructions live in prompts.

Where the Three src Files Fit

The src folder splits the work into three files. Each one has a narrow job, and the boundaries between them are what keep the code manageable. The comparison below lists each file and its responsibility as described in the source.

File	Responsibility	Stage of the work
main.py	Receives contracts, calls models, and returns memos	Entry point and model call
parse.py	Extracts text and sections from the contract	Reading the document
format.py	Shapes structured output into the memo format	Producing the finished memo

The three src files and their narrow responsibilities

Following One Contract Through the Layers

Now trace a single contract from the moment it enters the system to the moment a memo comes out. The source describes this path: a contract file goes in, its text and sections are extracted, the model is called and returns a structured result, and that result is formatted into a memo. The diagram below shows the sequence of responsibilities. It is not a picture of the code's internal function calls.

What happens to the contract after it is loaded, and which layer handles each step on the way to the finished memo?

Two details from the source are worth holding onto as you follow the path. First, the model is instructed to fill a structured output schema, so its answer arrives in a predictable shape rather than as free text. Second, a deterministic Python layer takes that structured output and produces the memo, so the final formatting does not depend on the model's wording. If a memo's layout is wrong, the cause is more likely in the formatting layer than in the model's reasoning.

What do you think happens?

Before the model is called, what has to happen to the contract?

The model reads the raw PDF bytes directly
The text and sections are extracted from the document
The memo is formatted in advance

Reveal answer

Answer:

The text and sections are extracted from the document

The source says text has to come out of the document first, and then the model does its reasoning. The extraction layer, parse.py, is the step that prepares the model's input.

Why a Small Layout Makes Debugging Easier

A common way to build an AI tool is to stack layers of libraries on top of the model: an agent framework to manage steps, an orchestration library to coordinate them, and a vector store to search documents. This code does none of that. The source states plainly that there is no agent framework, no orchestration library, and no vector store.

Instead, the whole slice runs on direct API calls to the model. Your code sends a request, the model fills in the structured output schema it was instructed to use, and ordinary Python code handles the rest. When you trace a problem, you follow plain function calls and plain data, with no hidden machinery between them.

The practical lesson is that a system small enough to read quickly is also easier to debug. Each layer has one job, the prompts are plain text, and no framework hides the sequence of steps. That is why the folder layout is useful: it tells you which layer owns a behavior, so you can inspect that layer first.

Choosing Where to Look First

The source's key instruction for debugging is this: when a memo is wrong, use the layout to match the symptom to a responsibility, and inspect the most likely layer first. The flowchart below turns that instruction into a sequence of questions. It is a generated study aid built from the responsibilities the source describes, not a procedure the source spells out step by step.

If the memo has a specific symptom, which layer or file should I check first, and what do I rule out next?

Worked example

A memo lists the wrong appendix

A reviewer reports that the memo's section summary omits an appendix that is clearly present in the contract file. Which layer should you inspect first?

1
Identify the symptom
The contract's content is present in the input, but the memo does not reflect it. The problem concerns finding sections and appendices in the document.
2
Match the symptom to a responsibility
The source assigns text extraction and section detection to parse.py. An appendix that is never found is a detection problem, so parse.py is the most likely layer.
3
Rule out the next layer
If parse.py did find the appendix, the model received it, and the question shifts to the model call and prompts. Only after that would you look at format.py.
4
Check the run record
The traces folder holds per-run JSON dumps, so the saved output of this run can show whether the appendix reached the model.

Result

Inspect parse.py first, because the symptom is a section that was not found. The layout tells you where to start without reading all of src.

Mistakes When Locating Responsibility

Common mistake
Assuming the model is responsible for every wrong output
What it looks like
The memo's layout is off, and the learner starts editing the prompts.
Why it fails
The source says a deterministic Python layer produces the memo, so the final formatting does not depend on the model's wording.
Do this instead
Check format.py when the problem is in how the memo is shaped, and check the prompts only when the model's answer itself is wrong.
Common mistake
Reading every file before forming a hypothesis
What it looks like
A learner opens all three src files and the evals folder before deciding anything about a missing section.
Why it fails
The layout exists so that you can find the responsible part without reading everything.
Do this instead
Match the symptom to a responsibility first, then inspect the most likely layer.
Common mistake
Treating the file names as the full description of the pipeline
What it looks like
A learner assumes main.py runs first because it has the lowest name in the alphabet.
Why it fails
The source notes that the file names do not spell out the order; the descriptions do.
Do this instead
Use the job descriptions to trace the path: text extraction, then the model, then formatting.
Common mistake
Expecting hidden framework behavior to explain a bug

Practice: Locate the Layer

MEDIUM

A reviewer reports that the memo's formatting puts the summary in the wrong position, but the extracted text and the model's structured result both look correct. Using the project layout, name the file you would inspect first, and say which layers you would rule out and why.