Increased Costs and Latency of AI Judges

Imagine you are evaluating a chatbot. For each answer it gives, you use an AI judge to score how faithful the answer is to the context it was given, meaning how well the answer sticks to that context. You run the evaluation on Monday and get a number. You run it again on Tuesday on the same answer and get a different number. Nothing in the chatbot changed. So what does the number actually mean?

This article follows one chain of reasoning. First, you will trace what happens when the same judge scores the same input more than once. Second, you will see why two tools that share a criterion name still produce numbers that cannot be lined up. Third, you will trace how a change on the judge side can masquerade as a change in the application. Finally, you will practice the checks that should happen before you trust a score trend.

A trustworthy evaluation method should give consistent results. If you score the same answer twice and get different numbers, you cannot be sure what the number means, and you cannot reproduce a result that someone else reports. This is the reproducibility problem. The source explains it directly: because AI judges are probabilistic, the same judge can score the same input differently when its prompt changes or when it is run again.

Read the diagram as a sequence. The input never changes between runs, and neither does the criterion. What varies is the judge's output on each pass. The spread between the highest and lowest scores is the amount of noise you would have to see through before a real difference between two systems could be claimed. A single run tells you one draw from that spread, not the judge's verdict.

Back to the chatbot. Two tools you are considering, MLflow and Ragas, both offer a built-in faithfulness criterion. It is tempting to assume that a faithfulness score is a faithfulness score, so that results from both tools could sit in one report. The source identifies this assumption as the misconception to correct: a shared criterion name does not guarantee a shared measurement.

The reason is that criterion names are not standardized. Many human-designed metrics follow a standard definition, but AI-as-a-judge metrics generally do not. The open source tools MLflow, Ragas, and LlamaIndex all include a built-in criterion called faithfulness, which asks how faithful a generated output is to the given context. Their instructions to the judge and their scoring systems differ from one another, so their scores cannot be compared directly.

Read down each column. The MLflow prompt asks for a graded judgment about the output as a whole. The Ragas prompt works statement by statement, so a single unsupported claim changes the verdict for that statement. The LlamaIndex prompt is lenient about partial support, so a single supporting passage is enough for a YES. Each design answers a slightly different question, even though all three are labeled faithfulness. The differences go beyond the number of points on a scale, so you cannot fix the mismatch by simply rescaling one tool's numbers.

For the chatbot, this means a faithfulness score from MLflow and one from Ragas are answers to two different questions about the same answer. A report that mixes them would be combining measurements that were never defined the same way. If you want to compare the tools, you need to run them on the same data and report each result with its tool and configuration attached, rather than treating the labels as interchangeable.

The third problem is about time. An application evolves, but its evaluation ideally stays fixed so that metrics can track the application's changes from one period to the next. The difficulty is that AI judges are themselves AI applications, so they can also change over time. Their prompts get edited, their underlying models get swapped, and their behavior shifts. When that happens, a change in the scores may reflect the judge rather than the application.

Consider a coherence score that rose from 90% last month to 92% this month. The natural reading is that the application became more coherent. That conclusion holds only if the judge is exactly the same in both months. The judge's prompt might have been replaced with a slightly better-performing version, or a small edit to the prompt might have made the judge more lenient. In either case, the 2-point rise could come entirely from the judge.

Trace the two columns. The application version stays the same, so the only variable that changed between the periods is the judge prompt. The 2-point rise is therefore not evidence about the application until the judge is shown to be the same. This is the pattern to watch for: a score trend moves, the application did not visibly change, and the judge did.

Because a trend can come from the judge as easily as from the application, it helps to work through a fixed sequence of checks before accepting it. The sequence below starts with recording what produced each score, then confirms the judge setup is the same across the periods being compared, and only then asks whether the movement is larger than the run-to-run variation you would expect anyway.

Read the flowchart as a gate. The first decision asks whether the judge and prompt are the same in both periods. If they are not, the shift cannot be credited to the application without further evidence. If they are, you still need to confirm that sampling and other judge settings did not move, and that repeated runs on the same input show a spread narrower than the movement you are about to interpret. Only then does the trend deserve to be treated as a signal, and even then it remains provisional until the judge has been checked against answers with known correct verdicts.

A strong answer names the judge prompt as the changed variable, refuses to credit the application until the judge is shown to be identical across periods, and then describes the repeat-run check. It also notes that the score is only meaningful alongside the judge model and prompt that produced it.