Writing Rubrics That a Judge Can Apply Consistently

Imagine a legal team that must grade a large number of contracts against a set of criteria. The source describes exactly this situation. The grading effort would have stalled at the original forty-two cases without help. The deputy general counsel could grade five contracts an hour at most, while the team needed grading capacity in the hundreds per week, and that need grew as the prompt was iterated.

A human reviewer cannot keep up with that volume, so the team considered an LLM-judge: a model used to grade the contracts in place of a person. The obvious question is whether the judge can be trusted. Calibration is the process of answering that question with evidence rather than with hope. This article follows how the source team did it, and why the answer turned out to be different for different criteria.

Before looking at any numbers, it helps to follow what goes into a calibration run and what comes out. There are two sources of grades. The first is the handlabeled set, the contracts that the deputy general counsel had graded by hand. In the source, this set contained fifty contracts. The second is the judge's grades, produced by the LLM-judge for those same contracts. Calibration brings these two sets together so they can be compared one criterion at a time.

Notice that the judge is never asked to justify itself in isolation. Every judge grade is measured against a human grade on the same contract. The human-labeled set is the reference point, and the output is an agreement percentage for each criterion rather than a single overall score.

Agreement here means the degree to which the judge's grade matches the human grade for the same contract on the same criterion. Because the source reports agreement as a percentage, the team could compare criteria directly. A high agreement percentage on one criterion says nothing about another criterion, which is why the source keeps the results separate.

Calibration was not a single test at the end of the project. The source states that the judge took six iterations to get into useful range before it was useful at all. Each pass followed the same loop: check the judge's behavior against the handlabeled set, revise the grading instructions, and check again.

The loop has a clear structure. A criterion is measured. If agreement is not high enough, the criterion stays with the human reviewer, the rubric is revised, and the agreement is measured again. The same criterion can therefore pass through several rounds before a decision is made about it.

Once agreement is measured for a criterion, the team must decide where that criterion is graded. The flowchart below shows the decision the source describes. If agreement is high enough, the criterion can be graded by the judge. If not, the criterion stays manual while the rubric is revised and re-measured.

The source reports four criteria, and their results differ in a useful way. C1 is the hard-fail criterion, meaning that failing it fails the contract outright. Because of that role, the team needed the judge to agree with the human labels on C1 before trusting it there. Agreement reached ninety-four percent after two rubric revisions, which was high enough for the team to trust the judge on C1.

C2 shows what happens when agreement does not reach the bar. Agreement stalled at seventy-one percent, which the source describes as not high enough. The team kept C2 grading manual until the rubric stabilized, so the human reviewer continued grading that criterion while the judge was not yet trusted on it. A third pass, made once the rubric stabilized, brought agreement on C2 to eighty-six percent. The source does not describe the exact changes made to the C2 rubric.

C3 and C4 were different. Both calibrated cleanly from the first attempt because their instructions were crisp and mechanical. The judge matched the human labels without any rubric revisions. The source attributes this to the nature of the criteria themselves, not to any special tuning.

Reading the table this way shows the lesson of the source. A criterion can stall below the bar, the team can hold it back from automation, and it can later be brought into automated grading once the evidence supports it. Even C1 needed two revisions before its result was trusted, so calibration results should be read as the outcome of a process rather than as a property of the criterion alone.

The contrast is useful for planning. Criteria that depend on clear, checkable features tend to calibrate quickly, while criteria whose instructions leave more room for interpretation may need several rounds of rubric revision. The source does not claim a general rule here, so use this pattern as a hypothesis to test on your own criteria rather than as a guarantee.