Designing Eval Criteria for AI Products
A vague quality concern cannot drive improvement, because no one can say whether a change satisfied it.
Two Teams, One Complaint
Every AI product team eventually says something like 'the answers aren't good enough' or 'this feels off.' These statements are honest, and they usually point at something real. The trouble is that they cannot be checked. One person reads a response and nods, another reads the same response and frowns, and neither can show the other who is right. When a quality concern cannot be checked, the team cannot tell whether a change made things better, worse, or nothing at all.
What do you think happens?
A team says its answers are not good enough, then changes the prompt. Can its eval say whether the change helped?
Reveal answer
No, because nobody can say whether the change satisfied a vague concern.
A vague concern gives no way to check an output. Without a check, a team cannot tell whether a change improved things, made them worse, or changed nothing.
This article covers the step that must come before any eval can be trusted: taking a vague quality concern and turning it into a measurable criterion. This conversion is the dividing line between two kinds of engagement. One kind improves its product over time. The other spends its effort debating whether the product is improving at all.
Why a Vague Concern Stalls Progress
Follow what happens when a team tries to improve a product against a concern it cannot measure. The team makes a change. It then looks at the outputs and asks whether the concern is now satisfied. If the concern is vague, the answer to that question is an opinion, and different people hold different opinions. The result is not a measurement. It is a disagreement.
The upper path is a loop that works. The team measures each change against criteria that stayed the same, keeps the change when the measure improves, and repeats. The lower path breaks at the measurement step. Each measurement is an opinion, and opinions are exactly what the team is arguing about, so no decision can be made from the result.
The same dependency explains why progress charts are only useful when the criteria are fixed. A chart that tracks week-by-week improvement is only meaningful if each week's measurement came from criteria that did not change while the product changed. Without measurable criteria, the chart would plot reviewers' moods rather than progress.
What Makes a Criterion Measurable
Objective evaluation converts a concern into a measurable criterion: an observable condition that different reviewers apply the same way.
The word 'measurable' does not mean that a number must appear on every line. It means the criterion is specific enough that two people reading the same output would reach the same verdict for the same reasons. A concern such as 'the answer is clear' only becomes objective once the team says what clarity looks like in terms a reviewer can check against the response.
The conversion is a sequence of decisions rather than a single insight. The team starts with the concern as stated. It then asks what would be seen in an output if the concern were satisfied. Finally, it writes that observable condition down as the criterion. The diagram below shows the before and after versions of this path.
The before and after versions describe the same underlying goal. Only the second one gives a reviewer something to check. The rewritten criterion is also narrower on purpose. It does not claim to capture every quality the team cares about, only the parts it can observe. That narrowing is what makes the eval usable.
Two Reviewers, One Verdict
A criterion is working when different reviewers reach the same verdict on the same output. When a criterion has an explicit definition and an explicit scale, the reviewers' verdicts converge. When the criterion is left to personal judgment, each reviewer fills the gap with their own standard. The comparison below uses an invented illustration to show the difference.
The explicit version does not guarantee that reviewers never disagree on hard cases. It does mean that when they disagree, the disagreement can be traced to a specific part of the output, rather than to two private and unstated standards. That is what allows a team to resolve the disagreement and improve the criterion.
Rewriting a Concern Step by Step
Turning 'The Answer Should Be Clear' Into a Criterion
A support assistant produces answers. The team's concern is that the answers should be clear. Rewrite this concern as a criterion that two reviewers can apply the same way. The example is generated for illustration.
- 1Start with the concern as stated
The concern is 'the answers should be clear.' It points at something real, but a reviewer cannot check it yet, because clear can mean different things to different people.
- 2Ask what would be seen if the concern were satisfied
Imagine a clear answer in front of you. A reasonable observable sign is that the first sentence directly answers the user's question. That is something a reviewer can look at without relying on a private sense of clarity.
- 3Write the observable condition as the criterion
Pass if the first sentence of the response directly answers the user's question. Fail if the first sentence is a greeting, a restatement of the question, or an answer to a different question.
- 4Check that two reviewers would agree
Give two reviewers the same responses. If both look only at the first sentence and apply the pass or fail rule, their verdicts should match. If they still diverge, the definition needs tightening, not more effort.
- 5State what the criterion does not cover
This criterion does not measure accuracy, tone, or length. Saying so openly keeps the team from reading a pass as proof that the answer is good overall.
The vague concern 'the answers should be clear' has become a criterion with an observable condition, a pass or fail verdict, and a stated boundary. Two reviewers can now apply it to the same output and reach the same verdict for the same reasons.
Mistakes That Keep an Eval Subjective
Common mistake Using undefined terms such as clear, good, or natural
What it looks likePass if the answer is clear.
Why it failsEach reviewer brings a private meaning of clear, so the same response can receive different verdicts.
Do this insteadReplace the term with an observable condition a reviewer can check against the response, such as whether the first sentence answers the question.
Common mistake Bundling several qualities into one criterion
What it looks likePass if the answer is accurate, friendly, and concise.
Why it failsA response can satisfy one part and fail another, so the single verdict hides which quality was actually missed.
Do this insteadSplit the qualities into separate criteria, each with its own observable condition and its own pass or fail verdict.
Common mistake Treating more effort as the fix
What it looks likeReviewers will agree once they try harder to be careful.
Why it failsObjectivity comes from the form of the criterion, not from the amount of effort. Two teams with equal effort can still produce very different evals.
Do this insteadRewrite the criterion until two reviewers applying it would reach the same verdict for the same reasons.
Common mistake Claiming the criterion captures all quality
What it looks likeThis criterion passing means the answer is good.
Why it failsA rewritten criterion captures only the parts that can be observed. Reading a pass as overall quality overstates what was measured.
Do this insteadWrite down what the criterion does not cover, so a pass is read as evidence for one specific condition only.
Common mistake Changing the criterion silently during the improvement loop
What it looks likeThe team loosens the definition after a disappointing week, then reports progress.
Why it failsIf the criteria change while the product changes, the measurements no longer compare like with like, and the progress curve plots reviewers' moods rather than progress.
Do this insteadKeep the criterion fixed while measuring. If it must change, make the change deliberately and record it, so that earlier and later measurements are not treated as the same measure.
Practice: Turn a Concern Into a Criterion
A team says that the summaries produced by its document tool feel off. Write one observable criterion for this concern. Include the condition a reviewer checks, the pass and fail verdicts, and one sentence stating what the criterion does not cover.
Hints
- Ask first what you would see in a summary if the concern were satisfied.
- Look for a condition that a reviewer can check without relying on a private feeling.
- If your criterion names two qualities, split it into two criteria.
Key Takeaways
- A vague quality concern cannot drive improvement, because no one can say whether a change satisfied it.
- Objective evaluation converts a concern into a measurable criterion: an observable condition that different reviewers apply the same way.
- Objectivity comes from the form of the criterion, not from the amount of effort a team spends.
- A rewritten criterion is deliberately narrower than the original goal, covering only the parts that can be observed, and it should stay fixed during an improvement loop.
- Common mistakes such as undefined terms, bundled qualities, and silent criterion changes keep an eval subjective and can be corrected by rewriting the criterion's form.