Evaluations as a Starting Point for Agent Work

An autonomous agent can look excellent in one environment and unreliable in another. The same model, given the same kind of task, may produce clean and useful changes in one codebase and wander or make careless errors in a different one. When this happens, the natural question is what the difference lies in. Part of the answer is the environment the agent works inside, and this article introduces the term used for that property: agent readiness.

Before reading on, consider what would have to be true of a codebase for an agent to make steady progress on it without a human checking every step. Hold your own answer in mind. The rest of the article tests it against the source's explanation, which centers on checks that the codebase itself can run.

The source names four familiar kinds of validation mechanism. A linter checks code against style and correctness rules. A type checker verifies that values are used consistently with their declared types. Tests run the code and compare its behavior with expected results. A security scan looks for known classes of vulnerability.

Each of these gives a clear pass or fail signal. That clarity is what makes the loop deterministic: the same check applied to the same code gives the same verdict, so the agent can rely on it. A validation loop is therefore not a judgment call. It is a fixed test that returns a verdict the agent can act on.

The related term validation density describes the presence of validation mechanisms such as linters, type checkers, tests, and security scans within a codebase. A codebase with many of these mechanisms covering many parts of the code has high validation density. A codebase with few or none has low validation density.

Density has two aspects that are easy to blur together. The first is how many mechanisms exist. The second is how thoroughly they cover the code. The source describes validation density as capturing how thoroughly those mechanisms cover the code, so a single check that touches only a small corner of a project contributes much less than several checks spread across it.

Agent readiness is closely tied to this idea. A dense set of validation loops is what the source means by a codebase that is ready for agents.

The easiest way to see what validation density does is to follow a single change. Suppose an agent edits a file. In a codebase with validation loops, that edit is not accepted directly. It is judged in sequence by each mechanism. The diagram below shows the path one agent step takes through the checks before it is accepted.

Each arrow in this pipeline is a point where the agent receives a verdict it can act on. A failed check gives the agent a concrete signal about what went wrong, and the agent can change its work in response. Only a change that passes every check moves on to acceptance. The loop back from the failure signal to the agent is what lets the agent correct itself while it works.

The contrast becomes clearer when two environments are compared directly, using the same kinds of checks. The diagram below sets a codebase with dense validation against one with sparse validation.

In the dense column, each step the agent takes can be checked by several independent signals. In the sparse column, the same step returns nothing. The agent has no external measure of whether its change is acceptable, so it has to proceed without one.

The source offers an analogy to how the underlying models were trained. Models of this kind were post-trained using dense reward, meaning they received feedback frequently rather than only at the end of a long task. The source observes that a validation-rich environment is dense reward by another name.

The parallel is about timing. Every validation loop returns a signal at a point in the work. That frequent, checkable feedback is what the model needed during training, and it is also what the agent needs while working. An agent that receives a verdict after each step can correct course early. An agent that receives no verdict until the end has to guess at every intermediate step.

The source makes a direct link between agent readiness and the autonomy an organization can hand to agents. The quality of the output of very long-running agent work is directly proportional to the degree to which that work can be validated. The more of the agent's work can be checked automatically, the more of it can be trusted without a human reviewing each step.

The source also implies that part of working with a customer is raising that customer's validation density, not only building the system itself. The evaluations written together with the customer are described as the beginning of the customer's agent readiness. Evaluations that define what correct behavior looks like are a starting point, and the checks that follow from them are what grow readiness over time.