Evaluating the Evaluation Pipeline

Many people who build an evaluation for an AI application treat it like a finished product. They write the criteria, design a scoring rubric, choose a handful of examples, and then assume the job is done. The evaluation becomes a fixed measuring stick that never needs recalibration. This is a common and understandable belief, but it breaks down quickly once the application is used by real people.

This lesson follows one idea through the whole pipeline: the evaluation is itself something you must keep checking. You will trace how a change in user behavior moves through the evaluation, why the criteria, rubric, and examples may each need revision, and why revision has to be balanced against the need for consistent results.

The reason the evaluation has to change follows directly from what it is for. An evaluation exists to tell you how well your application is doing against the goals that matter. When the people using the application shift what they need, or start interacting with it in new ways, the goals that matter shift too.

The evaluation criteria describe what counts as good behavior. If users start caring about something the criteria never mention, or stop caring about something the criteria still reward, the criteria no longer describe what users care about. The correction is to iterate: adjust the criteria to match the current situation rather than holding on to the criteria you wrote at the start.

Read the loop as a cycle, not a one-time fix. Each pass through it asks the same question: do the current criteria still describe what counts as good for the people using the application today? Whenever the answer is no, the pipeline moves on to revision and then measures again.

Iteration has three concrete levers. Each can be pulled independently, and a single change in user needs may call for several of them at once. The diagram below shows how the three parts sit inside the pipeline. Every part feeds the AI judges, and the judges' scores end up in the results.

Because changes to any of the three parts pass through the judges and show up in the results, a change to one part changes the outputs. This is exactly why changes need to be managed carefully. The three parts are examined one at a time below.

Notice that the team did not discard the original accuracy criterion. Iteration adjusts the criteria to match the current situation, and that can mean adding new criteria alongside existing ones. The aim is an evaluation that reflects what users care about now.

Revision alone is not enough. Iteration has to coexist with a reasonable level of consistency. An evaluation that changes constantly cannot reliably guide application development, because the people reading its results cannot tell what caused a difference between rounds.

Consider what a score change can mean. It might come from a change in the application, which is what you want to learn about. It might come from a change in the evaluation itself, such as a new rubric or a different judge configuration. If both happened at once and nothing was recorded, the score change cannot be attributed to either one.

In the left column, the rubric and the scores both changed, so a shift in results cannot be assigned to one cause. In the right column, the rubric and examples were held fixed and only the judge changed, so the difference can be traced to the judge. This is the sense in which a comparison is valid: you know which variable differed, and everything else you could have logged is the same.

The practical tool for keeping revisions interpretable is experiment tracking. Experiment tracking means logging every variable that could change during an evaluation process. When you compare two rounds of evaluation, the log shows exactly what differed between them. Without a record, you are left guessing whether a difference came from the application, the data, the rubric, or the judge's own configuration.