The Eval Suite from Chapter 9
The first-commit repository layout has a README, a prompts folder, a src folder with three Python files, an evals submodule, and a traces folder.
Why Start with the Layout
When an AI system behaves unexpectedly, the first useful question is rarely about the model. It is about where in the project the behavior is produced. A contract review system is built from several cooperating pieces: prompts that tell a model what to do, code that reads the contract, code that calls the model and returns a memo, and code that shapes the output into the format a firm expects. The repository layout is the map that tells you which piece lives where.
This article uses the repository as it stood after its first commit. The source describes that layout as almost embarrassingly simple, and that simplicity is useful for learning. Every directory and file in it has a stated purpose, so you can practice tracing a result back to the place that produced it. By the end you should be able to read the layout, assign each Python file its job, and choose which file to open first when a memo looks wrong.
Reading the Top-Level Folder
The top-level project folder is named contract-review-v1. Inside it sit a README, a prompts folder, a src folder, an evals folder, and a traces folder. Read the root as the project as a whole. It holds the documentation, the instructions given to models, the source code, the test material, and the stored run records.
The diagram shows containment. The README, prompts, src, evals, and traces items all sit directly under the root, and the three Python files sit inside src. The evals folder is the one holding the eval suite from Chapter 9. The traces folder is where stored run records live, which makes it the place to look when you want evidence of what a past run actually did.
The root and the folders beneath it form the first level of organization. Prompts and src form the second. Inside prompts, the files are separated by job. The source names a pre-pass that lists sections, a main redline prompt, and a playbook that holds the firm's standard positions. Inside src, the code is separated by stage. Recognizing both levels helps you decide whether a problem is about instructions or about code.
Three Files, Three Responsibilities
The src folder holds the code, and its three Python files are described by their roles rather than their contents. The difference matters when you debug. A file's role tells you what it should be responsible for, so when output is wrong you can ask which responsibility was violated.
main.py receives contracts, calls models, and returns memos. It is the entry point of the review flow: it takes the contract in, makes the model call, and hands back a memo.
parse.py handles text extraction and section and appendix detection. It is responsible for turning the incoming contract into usable text and identifying its sections and appendices before the model is involved.
format.py converts model output into the firm's memo format. It is responsible for the final shape of the result, not for what the model concluded.
Each file has one job. parse.py owns what the model reads, main.py owns the call to the model and the return of the memo, and format.py owns how the answer looks when it reaches the firm. Keeping these responsibilities separate means a wrong result can be traced to the stage that owns it.
Following a Contract Through the Pipeline
The three roles suggest a pipeline. A contract arrives as a PDF or Word file, its text and sections are extracted, the model is called and returns structured output, and that output is formatted into a memo. Read the diagram as a set of handoffs. Each handoff is a point where a wrong result can enter the system, and each handoff has an owner.
Before the model is involved, the contract text and section structure come from parse.py. After the model is involved, the structured answer is turned into a memo by format.py. Knowing which side of the model call a problem sits on is often enough to narrow the search to one file.
Locating the Stage That Owns a Symptom
A reviewer reports that a memo lists the wrong section numbers for a contract, but the model's analysis of the clauses themselves looks reasonable. Which file owns the stage most likely responsible?
- 1Identify where the symptom appears
The wrong section numbers come from the document structure, which is the output of section detection.
- 2Match the symptom to a responsibility
parse.pyhandles section and appendix detection, so a section-structure problem belongs to its stage. - 3Check whether the model is the cause
The clause analysis looks reasonable, which suggests the model received and reasoned about the right content. That points away from the model call and toward the stage that prepares the input.
Inspect parse.py first. Its role owns section detection, so wrong section numbers are a signal to check that stage before blaming the model or the memo format.
Where the Prompts and Evals Fit
The README holds documentation, and the prompts folder holds the instructions given to models. Those instructions are not code in src, so a problem in what the model was told to do is a different kind of problem from a problem in how the code reads or formats the contract. The prompts folder separates its files by job: a pre-pass that lists sections, a main redline prompt, and a playbook that holds the firm's standard positions.
The evals folder holds the eval suite from Chapter 9. The source notes that it is linked into this repository as a git submodule, so that improvements flow both ways. A git submodule is a way of linking one repository inside another. In this project, that link means the evaluation material can be improved from the review system's repository and the improvements remain connected to the original suite.
When a result is wrong, separate analysis problems from layout problems. Map the symptom to the file whose role owns that stage.
Locating a Divergent Memo
Suppose a memo differs from the result you expected. Use the layout to decide where to look. Start by asking whether the problem sits before the model is involved, after it, or in the instructions. The flowchart below turns that reasoning into a sequence of checks.
A memo lists correct clause text but places a finding under the wrong heading in the firm's format. Using the flowchart, which file do you inspect first, and why is that the stage that owns the symptom?
Hints
- Ask first whether the input text and sections were wrong. The clause text is correct, so that branch is less likely.
- The problem is in the shape of the output, which belongs to the stage that converts model output into the firm's memo format.
Mistakes When Reading the Layout
Common mistake Blaming the model for every wrong memo
What it looks likeA memo has a misplaced heading, and the reviewer rewrites the prompt.
Why it failsThe source says the first useful question is almost never about the model. A layout problem is owned by the code that formats output, not by the model's reasoning.
Do this insteadDecide first whether the symptom is an analysis problem or a layout problem, then map it to the owning file.
Common mistake Treating parse.py and format.py as interchangeable
What it looks likeLooking in format.py for a wrong section number.
Why it failsparse.pyhandles section and appendix detection, whileformat.pyconverts model output into the memo format. They own different stages.Do this insteadMatch section and text problems to parse.py, and match memo shape problems to format.py.
Common mistake Assuming the diagram is the actual call order
What it looks likeBelieving main.py always calls parse.py before a particular step because the pipeline shows it that way.
Why it failsThe source names each file's job but does not spell out the exact call sequence.
Do this insteadWhen the order matters, trace the running code to confirm the real sequence of calls.
Common mistake Forgetting that the eval suite is a linked submodule rather than a plain folder
Common mistake Confusing the traces folder, which stores run records, with the evals folder, which holds the eval suite
Key Takeaways
- The first-commit layout has a README, a prompts folder, a src folder with three Python files, an evals submodule, and a traces folder, and each entry has a stated purpose.
main.pyreceives contracts, calls models, and returns memos;parse.pyhandles text extraction and section and appendix detection;format.pyconverts model output into the firm's memo format.- The three roles suggest a pipeline from contract to memo, with
parse.pybefore the model call andformat.pyafter it, though the exact call order must be confirmed by tracing the code. - When a memo diverges from expectations, separate input and analysis problems from layout problems, then inspect the file whose role owns that stage.
- The eval suite is linked as a git submodule so that improvements flow both ways.