Evaluating AI Responses with AI Judges
AI judges are probabilistic, so the same judge can score the same input differently when its prompt changes or when it is run again, which undermines reproducibility.
A Score That Moved Without a Code Change
Imagine you run evaluations on a chatbot. Each month you use an AI judge to score every answer for faithfulness, meaning how well the answer sticks to the context it was given. One month the average score rises, and the team wants to credit the chatbot's improvements. Before you do, you need to ask a harder question: did the chatbot change, or did the thing doing the scoring change? AI judges are themselves AI applications, so they can change too. This article follows a single score through the evaluation process so you can see where the judge can move the number and what to check before believing a trend.
What do you think happens?
The chatbot's answers are identical on Monday and on Tuesday, and you send the same answer to the same AI judge both days. What do you expect the score to do?
Reveal answer
It can differ between the two runs, even though the input is the same.
AI judges are probabilistic, like other AI applications. Their outputs are not fixed functions of their inputs in the way an arithmetic formula is, so the same judge can score the same input differently when it is run again.
Tracing One Score Through the Judge
Follow one answer from the application to the trend line. The application produces a response. That response is handed to the judge, which is a model running with a specific prompt. The judge produces a verdict, which becomes a score. Many scores, collected over time, form the trend that the team reads. Each arrow in this path is a place where the number can shift, so the path is worth tracing step by step before interpreting anything at the end.
Variability can enter at the judge step. The prompt and the model both sit inside the judge, and either one can change between runs or between months. The application's response is only one input to that step. If you want the trend to measure the application, you must control or record the judge side of the pipeline as carefully as you control the application side.
Why the Same Input Gets Different Scores
An AI judge is an AI application that scores the output of another AI application against a criterion. Because it is probabilistic, the same judge can score the same input differently when its prompt changes or when it is run again. This is the root of the reproducibility problem: a trustworthy evaluation method should give consistent results, and if the same answer receives different numbers, you cannot be sure what a number means or reproduce a result someone else reports.
The contrast with an arithmetic formula is useful here. If you compute a fixed function of an input, you get the same output every time you compute it. An AI judge does not behave that way. Its output is produced by a probabilistic process, so the score is one draw from a range of possible verdicts rather than a fixed property of the answer. That does not make the judge useless, but it means every score needs context: which judge produced it, and under which prompt.
Testing whether a judge is stable enough to use
You want to know whether your AI judge gives reproducible faithfulness scores for a set of answers. You have ten answers where a human has already decided the correct faithfulness level, and you plan to score each answer several times.
- 1Score the same answer more than once
Run the same judge with the same prompt on the same answer several times. If the scores differ from run to run, the judge is not reproducible for that answer, and you cannot be sure what a single score means.
- 2Compare the repeated scores to the known correct answer
Check whether the judge's scores agree with the human-decided answers. A judge that is consistent but disagrees with the known answers is repeating a mistake reliably, which is stability without accuracy.
- 3Record the judge model and prompt with every score
Store which judge model and which prompt produced each score. Without this record, a later difference in scores cannot be traced to its cause.
A judge earns trust for this use only when its repeated scores are stable and its scores also match the known correct answers. Stability alone is not enough.
Same Criterion Name, Different Measurement
Back to the chatbot. You are considering two tools, MLflow and Ragas, and both offer a built-in faithfulness criterion. It is tempting to assume that a faithfulness score is a faithfulness score, so that results from both tools could sit in one report. The source pack names this as the misconception to correct: a shared criterion name does not guarantee a shared measurement. Criterion names such as faithfulness are not standardized, so tools use different prompts and scoring scales, and their scores cannot be compared directly.
The three open source tools MLflow, Ragas, and LlamaIndex all include a built-in criterion called faithfulness, which asks how faithful a generated output is to the given context. Their instructions to the judge differ, and so do their scoring systems. The differences go beyond the number of points on the scale. Each design answers a slightly different question, even though all three carry the same label.
| Tool | Criterion label | How the judge works | What a YES or strong score means |
|---|---|---|---|
| MLflow | Faithfulness | Graded judgment about the output as a whole | The whole output is graded on how faithful it is to the context |
| Ragas | Faithfulness | Works statement by statement | Each statement is judged, so a single unsupported claim changes the verdict for that statement |
| LlamaIndex | Faithfulness | Lenient about partial support | A single supporting passage is enough for a YES |
All three tools use the label faithfulness, but each asks the judge a different question. Scores from these designs measure different things.
When the Judge Changes, Not the Application
Evaluation is most useful when it stays fixed while the application evolves, so that metrics can track the application's changes from one period to the next. The problem is that the judge can change too. Its prompt may be edited, its underlying model may be swapped, and its behavior may shift. When that happens, a change in the scores may reflect the judge rather than the application. The source pack gives a concrete case: a coherence score that rose from 90% last month to 92% this month. The natural reading is that the application became more coherent. That reading holds only if the judge is exactly the same in both months.
In the diagram, the judge prompt changed between the two months, and the score changed with it. The source describes two ways this can happen: the prompt might have been replaced with a slightly better-performing version, or a small edit might have made the judge more lenient. Either way, the 2-point rise is evidence about the judge and the application together. A reader who only sees the number cannot separate the two causes.
The problem gets worse when different teams own the application and the judge. The judge team may update the judges without telling the application team. The application team then sees a shifted score and attributes it to its own work, when the real cause lies in the judge. Keeping a record of which judge and which prompt produced each score is what makes such shifts traceable.
Checks to Run Before Trusting a Trend
Before a score trend is treated as a real change in the application, run the checks in a fixed order. Each check either rules out a cause of the shift or sends you back to investigate. The flowchart below shows the sequence. The steps that follow it name the same checks in words so you can apply them to any trend.
- Confirm that the judge model and judge prompt are the same in both periods. If either changed, the score shift may come from the judge.
- Rerun the judge on the same inputs to check that it gives reproducible scores. Scores that vary between runs cannot be read as a stable signal.
- Check the judge against answers whose correct scores are already known. A stable judge that disagrees with known answers is repeating a mistake.
- Confirm that every score in the trend was produced by the same criterion definition. Do not combine scores from different tools as if they measured the same thing.
- Only after these checks pass, attribute the trend to a change in the application.
Your team's faithfulness score for a support chatbot rose from 84% to 89% after a judge prompt was edited last week. A colleague says the chatbot improved. Using the checks from this article, explain what you would verify before agreeing, and say which check would be the most informative first step.
Hints
- Start with whether the judge changed. The prompt was edited, so the judge is not the same in both periods.
- Ask whether the judge is consistent on repeated runs and whether it agrees with answers whose correct scores are known.
- A rise from 84% to 89% after a prompt edit is consistent with either an application change or a judge change. What evidence separates them?
Mistakes Beginners Make with AI Judge Scores
Common mistake Treating a stable score as a correct score
What it looks likeA judge returns the same faithfulness score for an answer every time, so the team concludes the score is trustworthy.
Why it failsConsistency is not accuracy. A judge can repeat the same mistake reliably.
Do this insteadCheck the judge's scores against answers whose correct scores are already known, not only against its own repeat runs.
Common mistake Assuming a shared criterion name means a shared measurement
What it looks likeFaithfulness scores from MLflow and Ragas are placed on one chart as if they came from the same method.
Why it failsCriterion names are not standardized. The tools use different prompts and scoring scales, and their scores cannot be compared directly.
Do this insteadKeep one tool's criterion definition for a comparison, or re-score the answers with a single defined method.
Common mistake Crediting the application for every score shift
What it looks likeCoherence rises from 90% to 92% after a judge prompt edit, and the team reports that the chatbot became more coherent.
Why it failsA change in the judge or its prompt can look like a change in the application. The conclusion holds only if the judge is the same in both periods.
Do this insteadRecord the judge model and prompt for every score, and confirm they match before reading a trend as an application change.
Common mistake Dropping the judge details from the record
What it looks likeScores are saved without noting which judge model or prompt version produced them.
Why it failsWithout the record, a later shift cannot be traced to its cause, and another team cannot reproduce the result.
Do this insteadStore the judge model and prompt alongside each score.
Key Takeaways
- AI judges are probabilistic, so the same judge can score the same input differently when its prompt changes or when it is run again, which undermines reproducibility.
- Consistency is not accuracy: a stable judge can repeat the same mistake, so check its scores against known correct answers.
- Criterion names such as faithfulness are not standardized. MLflow, Ragas, and LlamaIndex use different prompts and scoring scales, so their scores cannot be compared directly.
- A change in the judge or its prompt can look like a change in the application, so tie every score to the judge model and prompt that produced it, and run the checks before trusting a trend.