Designing the Contract Review System
The first-commit repository layout has a README, a prompts folder, a src folder with three Python files, an evals submodule, and a traces folder.
Why a Wrong Memo Points to a Location
When an AI system behaves unexpectedly, the first useful question is almost never about the model. It is about where in the project the behavior is produced. A contract review system is not one piece of code. It is several cooperating pieces: prompts that tell a model what to do, code that reads the contract, code that calls the model and returns a memo, and code that shapes the output into the format a firm expects.
The repository layout is the map that tells you which piece lives where. This article uses the repository as it stood after its first commit. The source describes that layout as almost embarrassingly simple, and that simplicity is useful for learning. Every directory and file in it has a stated purpose, so you can practice tracing a wrong result back to the place that produced it.
Before reading further, imagine the memo that comes out of the system is formatted correctly but states the wrong conclusion about a liability clause. Write down which of the following you would check first: the way the memo is laid out, the way the contract text was read, or the wording of the model instructions. Then compare your answer with the rest of this article.
Hints
- Think about which stage produces the conclusion and which stage only arranges the output.
- The later sections describe the responsibility of each file in the pipeline.
Reading the Top-Level Layout
The top-level project folder is named contract-review-v1. Inside it sit a README, a prompts folder, a src folder, an evals folder, and a traces folder. The root holds the project as a whole. It contains the documentation, the instructions given to models, the source code, the test material, and the stored run records.
Notice the second level of organization. Inside prompts, the files are separated by job. There is a pre-pass that lists sections, a main redline prompt, and a playbook that holds the firm's standard positions. Inside src, the source code is separated by stage. The diagram shows the root-level items beside each other and the contents of prompts and src one level down. Only src contains the three Python files, so when you need code, you know to look in src first.
Three Files, Three Responsibilities
The src folder holds three Python files: main.py, parse.py, and format.py. The source describes each one by its role rather than by its contents, and that difference matters when you debug. A file's role tells you what it should be responsible for. When output is wrong, you can ask which responsibility was violated instead of reading every line of code.
main.py receives contracts, calls models, and returns memos. It is the file that sits between reading the contract and producing the final result.
parse.py handles text extraction and section and appendix detection. It is responsible for turning the contract file into text and identifying how that text is structured.
format.py converts model output into the firm's memo format. It is responsible for layout of the result, not for deciding what the result says.
| File | Single job | Typical question it answers | Stage relative to the model |
|---|---|---|---|
| main.py | Receives contracts, calls models, returns memos | Was the model called and was its result returned? | Around the model call |
| parse.py | Text extraction and section and appendix detection | Did the system read the contract text and its structure correctly? | Before the model is involved |
| format.py | Converts model output into the firm's memo format | Is the output arranged in the firm's memo format? | After the model is involved |
Each src file owns one stage of the contract review pipeline.
Following a Contract Through the Pipeline
The three roles suggest a pipeline. A contract arrives as a PDF or Word file. Its text and sections are extracted, the model is called and returns structured output, and that output is formatted into a memo. The diagram shows that order as implied by the roles. Read it as a set of handoffs. Each handoff is a point where a wrong result can enter the system, and each handoff has an owner.
Locating the owner of each handoff
A contract is passed through the review system. For each handoff in the pipeline, name the file that owns it and state what it hands forward.
- 1Start with the contract file
The contract arrives as a PDF or Word file. Nothing has been read yet, so no model has been involved.
- 2Assign the extraction handoff
Text extraction and section and appendix detection belong to
parse.py. Its output is the contract text and its structure, which moves toward the model call. - 3Assign the model handoff
main.pyreceives the contract, calls the model, and returns the result. Its output is the structured answer the model produced. - 4Assign the formatting handoff
format.pyconverts that model output into the firm's memo format. Its output is the memo itself.
parse.py owns the handoff before the model, main.py owns the model call and return, and format.py owns the handoff after the model. A problem in the text or sections points to parse.py, while a problem in the memo layout points to format.py.
Tracing a Wrong Memo to Its Owner
The source gives a clear rule for debugging: when a result is wrong, separate analysis problems from layout problems, and map the symptom to the file whose role owns that stage. The flowchart below turns that rule into a sequence of questions. It is a teaching aid built from the file roles in the source, not a procedure the source spells out step by step.
Read the decision path as a narrowing tool. If the contract text or its section structure is wrong, the search ends in parse.py before the model is ever blamed. If the text was read correctly and the memo is arranged badly, the search ends in format.py. If neither of those is the cause, the remaining candidates are the model call and the instructions the model received, which is where main.py and the prompts folder come in. The source's point is that knowing which side of the model call a problem sits on is often enough to narrow the search to one file.
A memo lists the wrong appendix as the source of a payment term, and the memo's headings are in the firm's correct order. Using the repository layout and the file roles, decide which file to inspect first, and explain why the other two files are less likely to be the owner of this specific symptom.
Hints
- Ask whether the error is about which part of the contract was read, or about how the output is arranged.
- Appendix detection is named as part of
parse.py's job.
Mistakes When Reading the Layout
Common mistake Assuming the model is the first place to look when the memo is wrong
What it looks likeA memo cites the wrong section, and the learner immediately rewrites the prompts.
Why it failsThe source says the first useful question is about where in the project the behavior is produced, not about the model. The section may have been identified wrongly by
parse.pybefore the model ever saw it.Do this insteadCheck whether the text and section structure are right first, then decide whether the problem sits on the model side of the handoff.
Common mistake Treating format.py as the place where the answer is decided
What it looks likeA memo states the wrong conclusion, and the learner edits format.py to change the wording.
Why it failsformat.pyconverts model output into the firm's memo format. Its role is layout, so it does not own the substance of the conclusion.Do this insteadWhen content is wrong but the layout is right, look upstream toward the model call and the prompts.
Common mistake Reading the file roles as the exact order of calls
Why it failsThe source describes the roles and implies a pipeline, but it does not spell out the exact call sequence between the files.
Do this insteadTrace the running code to confirm the actual order of calls before relying on the implied sequence.
Common mistake Confusing the evals folder with the review pipeline
Why it failsThe evals folder holds the eval suite from Chapter 9, which is linked as a git submodule. It is test material, not one of the files that produces a memo.
Where Evals and Traces Fit
The evals folder holds the eval suite from Chapter 9. The source notes that it is linked into this repository as a git submodule, so that improvements flow both ways. A git submodule is a way of linking one repository inside another. In this project, that link means the evaluation material can be improved from the review system's repository, and the improvements remain connected to the original suite.
The traces folder is the stored run records. Together with the eval suite, it sits beside the pipeline rather than inside src. The source does not describe how the traces are produced or how they connect to individual memo results, so this article does not claim more than that. What the layout does show is that the pipeline files, the eval material, and the stored records are separate items at the root, each with its own stated purpose.
Checking Your Reading of the Layout
Without looking back, list the five top-level items in contract-review-v1, then state the one-job responsibility of each of main.py, parse.py, and format.py. Finally, name the file you would inspect first if the memo's conclusion about a clause is wrong but its headings and layout match the firm's format.
Hints
- The five top-level items are the README, prompts, src, evals, and traces.
- A conclusion is a substance problem, and the layout is correct, so think about the stage before the layout step.
Key Takeaways
- The first-commit layout of contract-review-v1 contains a README, a prompts folder, a src folder with three Python files, an evals submodule, and a traces folder.
main.pyreceives contracts, calls models, and returns memos.parse.pyhandles text extraction and section and appendix detection.format.pyconverts model output into the firm's memo format.- The three files form a pipeline of handoffs, and each handoff has one owner. Knowing which side of the model call a problem sits on often narrows the search to one file.
- When a result is wrong, separate analysis problems from layout problems and map the symptom to the file whose role owns that stage. Confirm the actual call order by tracing the running code.