Evals as the foundation of agent evaluation
Agent readiness is a measure of how many deterministic validation loops are present inside a codebase.
Same Agent, Different Results
A capable autonomous agent can look brilliant in one customer's environment and disappointing in another's. The model has not changed, and the task may look similar on paper. What differs is the environment the agent works inside. The surrounding codebase, not only the agent, shapes how much the agent can be trusted to do without supervision.
Keep one question in mind as you read: when an agent produces work, what is there to check that work against? The answer governs how much independence an organization can give the agent. This article uses that question as the thread that connects everything else.
Checks That Return a Verdict
Agent readiness is a measure of how many deterministic validation loops are present inside a codebase. The key word is measure. Agent readiness is not a yes or no property of a team or a tool. It is a degree, describing how much of the codebase is equipped with checks that run automatically and return a definite result. The more of these checks a codebase contains, the more readiness it has.
The source material names four kinds of checks that make a codebase dense: linters, type checkers, tests, and security scans. Each one examines code automatically and gives a verdict that does not depend on someone's opinion in the moment. Taken together, these checks form the validation loops that give agent readiness its meaning.
Density Measures Coverage
Validation density is the presence of those loops across the codebase, and it determines how much checking is available for an agent's work. Density is about coverage across the codebase, not about how good any single check is. A codebase with a few checks in one corner has low density even if those checks are excellent. A codebase where many parts of the code are covered by linters, type checkers, tests, and security scans has high density.
Counting coverage in two repositories
Repository A has a thorough test suite for its billing module and nothing elsewhere. Repository B has a linter, a type checker, tests, and a security scan that together touch most of its modules. Which one has higher validation density?
- 1Ask where the checks are
Repository A's checks sit in one corner of the code. Repository B's checks reach across most of its modules.
- 2Apply the definition of density
Density is about coverage across the codebase. A strong check in one corner does not make the whole codebase dense.
- 3Compare the two
Repository B has many more parts of its code covered by automatic verdicts, so it has the higher validation density.
Repository B has higher validation density, even though Repository A's billing tests may be excellent.
Tracing an Agent Change Through the Loops
Follow one agent change through a dense set of loops. Each loop returns a definite verdict, and the change only moves forward when the checks pass. If a check fails, the change goes back for revision rather than continuing down the path. Where a loop is missing, the path has a gap, and nothing on that stretch of the codebase can push back on a bad change.
What Each Loop Contributes
- Linter: examines code automatically and returns a verdict that does not depend on opinion in the moment.
- Type checker: examines code automatically and returns a verdict that does not depend on opinion in the moment.
- Tests: examines code automatically and returns a verdict that does not depend on opinion in the moment.
- Security scan: examines code automatically and returns a verdict that does not depend on opinion in the moment.
- Together: the four kinds form the validation loops that give agent readiness its meaning.
Notice what each loop does not do. None of them replaces the agent's ability to produce work. Each one only gives the surrounding system a way to see whether a particular piece of work meets a standard, and to report that back in a form the agent can act on.
How Density Caps Autonomy
The source connects validation density directly to how much freedom an agent can be given. The quality of the output of long-running agent harnesses is directly proportional to the degree to which their work can be validated. In plain terms, the more you can check what an agent produced, the better the output you can expect from a long run, and the more freely you can let it run.
Autonomy is not granted because the agent is trusted in the abstract. It is granted in proportion to the checks that can catch a bad step before it compounds. Without validation, an organization has no basis for letting the agent go further, so it must keep a human close to every step.
Low Density Versus High Density
Consider the same agent given the same kind of task in two codebases. The first has a few checks in one corner. The second has linters, type checkers, tests, and security scans reaching most of its modules. The task is identical on paper, so the difference in behavior comes from the environment the agent is working inside.
What do you think happens?
Given the same task, in which environment is the agent more likely to be given wide freedom to run long tasks unsupervised?
Reveal answer
The codebase where linters, type checkers, tests, and security scans cover most of the code.
Autonomy is granted in proportion to the checks that can catch a bad step before it compounds. High coverage gives the organization a basis for letting the agent go further, while sparse coverage forces a human close to every step.
Mistakes That Overstate Readiness
Common mistake Judging readiness by one excellent check
What it looks likeA team has a rigorous test suite for one module and assumes the whole codebase is ready for autonomous agent changes.
Why it failsDensity is about coverage across the codebase. A strong check in one corner leaves most of the code without a verdict.
Do this insteadMap which parts of the codebase are covered by automatic checks and treat uncovered areas as gaps.
Common mistake Treating a flaky check as a real loop
What it looks likeA test that sometimes passes and sometimes fails for no clear reason is counted as validation.
Why it failsDeterministic means the same input yields the same verdict every time. An unstable verdict gives the agent nothing stable to push against.
Do this insteadFix or remove checks whose results are not repeatable before counting them toward readiness.
Common mistake Confusing capability with usable autonomy
What it looks likeBecause the model is strong, the agent is given long unsupervised runs in a sparsely validated codebase.
Why it failsValidation density sets the ceiling on how much capability can be used safely. Without checks, the organization cannot see whether the work is right.
Do this insteadSet the autonomy level by the validation coverage available, not by the model's abstract strength.
Common mistake Expanding autonomy faster than the checks can support
What it looks likeThe agent's permitted scope grows each week while the validation loops stay the same.
Why it failsThis creates a gap between what the agent is allowed to do and what anyone can verify.
Do this insteadExpand autonomy only as fast as validation loops can support it, and close the gap as part of the engagement.
Raising Density in a Customer Codebase
The source draws a practical conclusion for the forward-deployed engineer. Part of every engagement is raising the customer's validation density, not only building the system the customer asked for. The evals written together with the customer are the beginning of that customer's agent readiness.
- 1Map the codebase to see which parts already have linters, type checkers, tests, or security scans, and which parts have none.
- 2Write evals together with the customer so that the standard each check measures against is agreed and explicit.
- 3Add automatic checks to uncovered areas, favoring verdicts that are deterministic so the same input always yields the same result.
- 4Remove or repair flaky checks so that the loops the agent depends on give stable feedback.
- 5Raise the autonomy level only as the coverage of validation loops supports it, and keep the gap between permitted actions and verifiable actions small.
Practice: Predict the Agent's Path
A team has a linter, a type checker, and tests, but only the tests cover the payments module. An agent proposes a change to the payments module that passes its tests but would fail a security scan the team never configured. Before reading further, predict what happens to the change as it moves through the loops, and state how much autonomy the team should grant the agent for work in this codebase.
Hints
- Ask which loops actually exist for the payments module, and which loops are absent.
- Remember that the path only moves forward when checks pass, and that a missing loop is a gap nothing can push back on.
- Think about whether the agent's output can be verified in the places where the stakes are highest.
Key Takeaways
- Agent readiness is a degree, measured by how many deterministic validation loops a codebase contains.
- Validation loops are cycles in which work is produced, checked automatically, and judged against a standard; the four check types named in the source are linters, type checkers, tests, and security scans.
- Density is about coverage across the codebase, and it sets a ceiling on how much autonomy an organization can responsibly grant an agent.
- Agent capability does not change between environments; the surrounding validation determines how much of that capability can be used safely.
- Raising a customer's validation density, starting with evals written together with the customer, is part of the forward-deployed engineer's work.