Designing Evaluation Criteria for Language Model Outputs

Imagine you ask a model to write a short product description. In the same request you tell it how long the answer should be, which words must appear in it, and how the answer should be laid out. The text may read smoothly and still miss some of those demands. It might meet the length limit but leave out a required word, or use the right words in the wrong layout.

This raises a practical question: how do you measure how well a model handles a request like this, rather than only judging whether the text sounds fluent? The answer depends on a decision that every evaluation has to make first. Before you can score anything, you have to decide what counts as success. Once that decision is made, you can ask whether success can be checked by a program or whether it requires a judge who reads the output.

The word follow hides a real design choice. Two questions are built into it. First, which instructions count as part of the test at all? Second, what does it mean for an output to satisfy one of them? A benchmark has to answer both before it can produce a score.

Different benchmarks do not agree on what instruction-following covers. Some look mainly at whether the output has the right shape. Others also check the content, the language style, and the tone. The two benchmarks named in the source illustrate this difference well.

Because the two benchmarks cover different territory, their scores are not directly interchangeable. A model that does well on a format-focused benchmark has shown one kind of ability. It has not necessarily shown that it can keep to a required tone.

IFEval concentrates on whether a model can produce output in an expected form. Zhou et al. identified 25 types of instructions that can be verified automatically. Examples include requirements about including certain keywords, meeting length limits, using a given number of bullet points, and returning JSON. These are called automatically verifiable instructions because the requirement has clear criteria that a program can test.

The defining feature is that a checker can decide the answer without asking anyone's opinion. Take an instruction to use a particular word in a sentence. A short script can search the output for that word. Whether the instruction was followed becomes a yes or no question with a mechanical answer.

The check above is a simple substring test. It needs no human reading, and running it twice on the same output always gives the same answer. That reliability is what makes an instruction automatically verifiable.

The grouping of these instructions matters for evaluation design. Each group probes a different ability. Some groups test whether the model tracks specific vocabulary, others test whether it controls the length or structure of its answer, and others test whether it switches language when asked. A model can be strong in one group and weak in another, so a single overall number can hide that pattern.

INFOBench, created by Qin et al., takes a broader view. In addition to format, it evaluates content constraints such as restricting a response to one topic, linguistic guidelines such as a requested dialect, and style rules such as a required tone. Many of these demands cannot be checked by counting words or searching for strings. Whether a passage is in an appropriate tone for a young audience is a matter of judgment.

The benchmark's authors did not give up on checking these demands. For each instruction they wrote a list of criteria, and each criterion was framed as a yes or no question about the output. Those questions can be answered by a human evaluator or by an AI evaluator.

The two kinds of instruction differ in what the checker needs. An automatically verifiable rule needs only the output and the rule itself, and a program can apply the rule. A rule that needs judgment needs a reader, either a human or an AI judge, who answers a question about the output. The diagram below sets the two verification processes side by side.

Because the automatic checker needs no reader, it can be run again and again on new outputs without fresh human effort. A judgment-based check uses human or AI evaluators, so the way the criteria are written has a direct effect on the result.

The two checkers start from the same place, a model output, but they reach a score by different routes. The flowchart traces both routes. Follow the automatic path first, then the criteria path, and notice where they diverge.

On the automatic path the program produces a yes or no result for each verifiable instruction, and those results feed the pooled score. On the criteria path the criteria for each instruction are answered one by one, and the instruction is counted as followed only when all of its criteria are met. The two paths meet at the pooling step. The source states that overall scores depend on how criteria are pooled, but it does not give a single pooling formula, so the exact arithmetic depends on the benchmark.

The source recommends curating your own benchmark. The idea is to write the instructions your application really needs and attach criteria that define success for each one. The published benchmarks show you what the field has measured so far. Your own set shows you what your users will ask for.

The work usually moves from broad goals to specific checks. The hierarchy below shows one way to break an evaluation goal into categories, then into specific instructions, and finally into criteria that a checker or judge can answer.

Notice that the hierarchy ends at criteria that either a program can test or a judge can answer. A goal such as suits our product pages is too vague to score. Splitting it until each leaf is a yes or no question is what makes it evaluable. The example requirement from the source is a direct illustration: if your application needs a particular output format such as YAML, include a YAML instruction in your evaluation set and test the model on it.