Hard-Fail Criteria in Automated Evaluation

Imagine a legal team that must grade a large number of contracts against a set of criteria. The source describes exactly this situation. The grading effort would have stalled at the original forty-two cases without an LLM-judge. A judge is a model used to grade the contracts in place of a person.

The volume problem is concrete. The deputy general counsel could grade five contracts an hour at most, while the team needed grading capacity in the hundreds per week. That need grew as the prompt was iterated, because every change to the grading instructions meant more contracts had to be graded again. A human reviewer cannot keep up with that volume, so the team turned to a judge. The obvious next question is whether the judge can be trusted at all.

Calibration is the process of answering the trust question with evidence rather than with hope. A team that simply assumes its judge is accurate has no way to know when the judge is wrong. A team that measures the judge's grades against trusted human grades can say, with numbers, how often the two agree.

Calibration in the source ran against fifty contracts that the deputy general counsel had handlabeled. This handlabeled set is the reference point. Agreement is measured against those human grades, so the judge is never asked to justify itself in isolation. Its grades are always compared with a trusted standard.

Calibration is not a single test run at the end of a project. The source states that the judge took six iterations to get into useful range. The team checked the judge's behavior, revised the grading instructions, and then checked the judge again. The diagram below shows the order of these steps. Each handlabeled contract is graded by the judge, the judge's grades are compared with the human labels, and the agreement measurement feeds the trust decision. When agreement falls short, the rubric is revised and the judge is run again.

C1 is the hard-fail criterion in the source. A hard-fail criterion is one whose failure fails the contract outright, regardless of how the other criteria turn out. Because of that role, the team needed the judge to agree with the human labels on C1 before trusting it there. The source reports that agreement on C1 reached ninety-four percent after two rubric revisions, and that this was high enough for the team to trust the judge on C1.

Trust is decided criterion by criterion. The team did not ask whether the judge was good at grading contracts in general. For each criterion, they asked whether the judge agreed with the human labels on that specific criterion, and made a separate decision. The flowchart below shows the routing. A criterion's agreement result is checked against the team's bar. If agreement is high enough, the criterion is graded automatically. If not, it stays with manual review, and the rubric is revised and the criterion is checked again.

The source reports agreement separately for each criterion. Laying the results side by side shows why a single overall score would hide the important differences. The comparison below groups the results by how each criterion calibrated.

The source observes a pattern: criteria that depend on clear, checkable features tend to calibrate quickly, while criteria whose instructions leave more room for interpretation may need several rounds of rubric revision. The source is careful to say this is a pattern observed in one case, not a general law. Use it to plan effort, not to skip checking a criterion.