Automatically Verifiable Instructions

Suppose you ask a model to write a short product description, and in the same request you tell it how long the answer should be, which words must appear in it, and how the answer should be laid out. The model might satisfy some of those demands and miss others. A response can sound fluent and still fail two of the three requirements. That gap is the reason this article exists: fluency is not the same as following instructions, and measuring instruction-following needs its own method.

The practical question that follows is how you measure how well a model handles a request like this, not just whether the text sounds good. The answer depends on what you decide counts as success. Once that decision is made, you can ask a second question: can success be checked by a program, or does it require someone to read the output and judge it?

The word 'follow' hides a real design choice. Different benchmarks do not agree on what the capability covers. Some look mainly at whether the output has the right shape. Others also check the content, the language style, and the tone. Each benchmark therefore measures the particular notion of instruction-following it adopted.

The IFEval benchmark concentrates on whether a model can produce output in an expected form. Its authors, Zhou et al., identified 25 types of instructions that can be verified automatically. Examples include requirements about including certain keywords, meeting length limits, using a given number of bullet points, and returning JSON.

INFOBench, created by Qin et al., takes a broader view. In addition to format, it evaluates content constraints such as restricting a response to one topic, linguistic guidelines such as a requested dialect, and style rules such as a required tone. The two benchmarks are not in disagreement about the facts. They have drawn the boundary of 'instruction-following' in different places.

Consider an instruction to use a particular word in a sentence. A short script can search the output for that word. Whether the instruction was followed becomes a yes or no question with a mechanical answer. Length limits, bullet counts, and JSON validity work the same way: each can be tested by a program.

Many of the demands in INFOBench cannot be checked this way. Restricting a response to one topic, using a requested dialect, or matching a tone are not settled by counting words or searching for strings. Whether a passage has an appropriate tone for a young audience is a matter of judgment. These instructions need a human or model to interpret the output before they can be marked as followed or not.

Under an automatic checker, each instruction is paired with a program that tests it. The model produces a response, the program runs its test, and the result is a pass or a fail. The individual results are then combined into one aggregate score. The diagram below follows one response through that path.

When an instruction needs judgment, INFOBench's authors wrote a list of criteria for it. Each criterion is framed as a yes or no question about the output, and the questions can be answered by a human evaluator or by an AI evaluator. The judge reads the response and answers each criterion separately.

The key rule is strict. An instruction counts as followed under the criteria method only when all of its criteria are met. A response that satisfies two of three criteria does not get partial credit for that instruction under this rule. Overall scores then depend on how criteria are pooled across instructions, which is a separate design decision the evaluation must make explicitly.

The published benchmarks show you what the field has measured so far. Your own instruction set shows you what your users will ask for. The source recommends curating your own benchmark by writing the instructions your application really needs and attaching criteria that define success for each one.