Designing Rubrics for LLM-as-Judge Evaluation
Calibration checks an LLM-as-Judge by comparing its labels with domain experts' labels on a sample of roughly 50 to 100 outputs.
A Grader That Needs Checking
Imagine a team that uses a language model to grade thousands of answers produced by a product's assistant. Each answer receives a verdict such as acceptable or unacceptable, and those verdicts feed dashboards, release decisions, and reports. The obvious question is whether the grader's verdicts can be believed. The model may sound confident while disagreeing with the people who actually understand the domain. Calibration is the practice of answering that question with evidence before the grader's output is relied upon.
This article follows that question through a calibration run: how the sample is drawn, how experts and the judge are compared, what the 80 percent agreement threshold means, and why a change to the underlying model means the check has to be done again.
The Judge as a Measuring Instrument
Calibration checks an LLM-as-Judge by comparing its labels with domain experts' labels on a sample of roughly 50 to 100 outputs.
The key idea is that an LLM-as-Judge is itself a measuring instrument. Like any instrument, it needs to be checked against a reference before its readings are trusted. In calibration, the reference is the judgment of domain experts applied to a sample of real outputs. The judge's readings are only worth using once they have been shown to match that reference closely enough.
Following a Calibration Run
A calibration run moves through a fixed loop. The diagram below shows the steps as a flowchart. Read it from the top: a sample is drawn, experts and the judge label the same items, the two label sets are compared, and a decision is made. The important feature is the feedback arrow. When agreement falls short, the team does not simply accept the judge. It changes one of three levers and labels the sample again.
- 1Draw a sample of outputs from the material the judge will eventually evaluate. The source recommends roughly 50 to 100 outputs.
- 2Have domain experts label every item in the sample.
- 3Have the LLM judge label the same items, using the same rubric.
- 4Compare the two sets of labels to compute agreement.
- 5If agreement is above 80 percent, the judge is considered usable for that rubric, data, and prompt. If not, revise the rubric, the prompt, or the judge model, then run the labeling and comparison again.
Matching Judge and Expert Labels
For each output in the sample, there is one judge label and one expert label. The diagram lines up the possible pairs for a simple verdict with two values. A pair where the two labels match counts toward agreement. A pair where they differ counts as disagreement.
The 80 Percent Decision Rule
Once agreement has been computed, the decision is binary. The source sets the bar at above 80 percent. At or below the threshold, the judge is not yet ready. Above it, the judge is considered usable for the specific rubric, data, and prompt being tested. The table summarizes both sides of the line and what follows from each.
| Agreement result | Decision | What follows |
|---|---|---|
| Above 80 percent | Judge usable for this rubric, data, and prompt | The judge may be used under these exact conditions |
| At 80 percent | Judge not yet ready | Revise the rubric, prompt, or judge model, then run again |
| Below 80 percent | Judge not yet ready | Revise the rubric, prompt, or judge model, then run again |
How the 80 percent threshold sorts a calibration result
What do you think happens?
A calibration run finishes with agreement of exactly 80 percent. Is the judge usable?
Reveal answer
No, the judge is not yet ready.
The source sets the bar at above 80 percent, and it states that at or below the threshold the judge is not yet ready. A result of exactly 80 percent therefore falls on the not-ready side of the line.
Two runs on the same rubric
A team calibrates a judge on a sample of 50 outputs. The first run yields 40 items where the judge and experts agree. The team then revises the rubric and runs the labeling and comparison again on the same sample size. The second run yields 43 items of agreement. Is the judge usable after each run?
- 1Compute the first agreement rate
40 agreeing items out of 50 is 40 divided by 50, which is 0.80, or 80 percent.
- 2Apply the decision rule to the first run
The threshold requires agreement above 80 percent. An agreement of exactly 80 percent is at the threshold, so the judge is not yet ready. The team must revise the rubric, prompt, or judge model and run again.
- 3Compute the second agreement rate
43 agreeing items out of 50 is 43 divided by 50, which is 0.86, or 86 percent.
- 4Apply the decision rule to the second run
86 percent is above 80 percent, so the judge is considered usable for this rubric, this dataset, and this prompt. The result does not transfer to a different rubric or prompt.
The first run fails the threshold at exactly 80 percent. The second run passes at 86 percent, and the usable verdict applies only to the rubric, data, and prompt that were tested.
Mistakes When Reading Calibration Results
Common mistake Treating exactly 80 percent as passing
What it looks likeA run finishes at 80 percent agreement and the team deploys the judge.
Why it failsThe source requires agreement above 80 percent. At or below the threshold the judge is not yet ready.
Do this insteadCompare the result strictly against the threshold and revise and rerun when it is at or below 80 percent.
Common mistake Assuming a passing result applies everywhere
What it looks likeA judge passes calibration on one rubric and is then used on a different rubric without another run.
Why it failsCalibration is tied to a particular rubric, dataset, and prompt. A judge that agrees with experts on one rubric has not earned trust on another.
Do this insteadRecord which rubric, dataset, and prompt were tested, and calibrate again when any of them changes.
Common mistake Treating a failed run as a sign the whole approach has failed
What it looks likeA team abandons the judge after its first run falls short.
Why it failsThe source states the calibration loop repeats until the threshold is met, so iteration is a normal part of the process.
Do this insteadChange one lever at a time, whether the rubric, the prompt, or the judge model, then label the sample again.
Common mistake Assuming an earlier result still describes the running system
What it looks likeThe underlying model is updated, and the team keeps relying on last quarter's calibration result.
Why it failsA change to the underlying model invalidates the earlier result, because that result described a different system.
When a Model Change Resets the Judge
A calibration result describes one particular judge at one moment. If the underlying model changes, the earlier result no longer describes the system that is actually running. The source therefore requires re-calibration each time the underlying model changes. In the current cycle, the source notes that this happens roughly every quarter.
The diagram traces the lifecycle. A calibrated judge is not permanently trustworthy. A change to the model moves it out of the calibrated state, and it returns to that state only after a new calibration run meets the agreement threshold again. The judge's trust is therefore a property that has to be re-earned after each change, not a permanent label attached to the judge.
Checking a Calibration Result
A team calibrates a judge on a sample of 60 outputs under a fixed rubric and prompt. Agreement is 47 items out of 60. Decide whether the judge is usable under those conditions, state which side of the threshold the result falls on, and say what the team should do next. Then explain whether the judge would need re-calibration if the underlying model were updated next quarter.
Hints
- Compute the agreement rate as a percentage of the 60 items first.
- Check whether the rate is above 80 percent, not merely at or near it.
- Remember that a model update invalidates the earlier result.
Key Takeaways
- Calibration checks an LLM-as-Judge by comparing its labels with domain experts' labels on a sample of roughly 50 to 100 outputs, treating the judge as a measuring instrument that must be checked against a reference.
- A calibration run is a loop: draw a sample, have experts and the judge label the same items with the same rubric, compare the labels, and revise the rubric, prompt, or judge model when agreement falls short.
- The judge is usable only when agreement is above 80 percent, and that usability applies only to the specific rubric, data, and prompt that were tested.
- A change to the underlying model invalidates the earlier result, so the judge must be re-calibrated each time the model changes, roughly every quarter in the current cycle.