Evaluating AI Systems: A Shift in Engineering Mindset

Imagine two engineers who are each handed a system to build. The first receives a document that lists exactly what the system must do for every input it could receive. The second receives a collection of example cases, a rough description of what good performance looks like, and a warning that the bar may move as the project goes on. Both engineers will write code and run tests. Yet their daily work is shaped by different questions. The first engineer asks whether the system does what the document says. The second asks how well the system will perform across the situations it meets.

This article is about that difference. Moving from a specification to an eval set is more than a change in tooling or in the kind of artifact a team writes. It changes what the engineer is aiming for, and it changes how the engineer knows whether the aim has been reached.

Success under a specification is a yes-or-no judgment about correspondence. The behavior either agrees with what the specification asks for, or it does not. The engineer's question is therefore a matching question: does this system, for this input, do what the document says it should do?

Because the target is a fixed document, the engineer can in principle check every case the document covers against it. Each case is judged on its own. A single mismatch is a failure of the system against the specification.

The judgment is no longer a matter of exact agreement. It is a matter of how the system fares over the whole collection of cases the eval set represents, measured against the criteria the team has chosen. An eval set is a collection of cases, and the engineer's question is how the system performs across that collection rather than whether it reproduces a single stated answer.

The clearest way to see the contrast is to ask, for each kind of system, what counts as correct. Under a specification, correctness is a case-by-case match against a fixed document. Under an eval set, correctness is an aggregate result measured on criteria that may be revised. The same input can therefore produce a fixed, checkable output in one world and a range of acceptable outputs in the other.

The shift is best understood as a change in the engineer's relationship with the system. In the specification world, the engineer stands in a relation of matching. The job is to bring the system into line with a stated target and then confirm that the match holds. In the eval set world, the engineer stands in a relation of stewardship over an average. The job is to shape a system whose overall behavior is good, while accepting that individual cases will vary and that the yardstick itself may be revised.

Notice what this does to the engineer's daily questions. Under a specification, the engineer keeps asking whether the system matches the document. Under an eval set, the engineer keeps asking how the system performs across the cases it will meet, and whether the criteria still describe what the project needs. The technical work of writing code and running tests remains, but it is organized around a different aim.

Follow what happens when an eval set is used. A fixed collection of inputs is passed through the system. Each output the system produces is then scored against the team's criteria. The scores are gathered and read together, and the question the team asks is about the pattern across them. Variability enters at the point where the system produces its output for a given input, since a stochastic system does not have to give the same output every time. The criteria stay the same step to step within a given evaluation, but the team is free to revisit them between evaluations.

The example below is an invented illustration, written to show how the same engineering question is framed under each kind of target. It is not drawn from the source.

Engineers who are used to specifications often carry their habits into eval set work. The mistakes below are common when that carryover happens.