Building a Reliable Evaluation Pipeline for Open-Ended Applications

Many people who build with AI start from a comfortable assumption. Once a model is wrapped in an automatic scoring routine, its output seems to become an objective yardstick. You run the model, a judge assigns a number, and that number tells you whether the system improved. This assumption is the first thing a reliable evaluation pipeline has to examine, because foundation models break the simple picture in which a test has one right answer and a score can be read straight off the result.

The reason evaluation matters so much is tied to capability. The more capable a model becomes, the more serious its possible failures can be, which raises the importance of evaluating it well. The same capability, however, is what makes evaluation difficult. The rest of this article follows that tension step by step: first by looking at what an open-ended output is, then by sorting the evaluation methods, then by tracing how an AI judge's score moves through a pipeline and how comparative rankings are built.

A traditional machine learning model often answers a narrow question with a narrow output. Checking that output against an expected value is therefore straightforward: the test has a fixed target, and the model either hits it or it does not. A foundation model is open-ended. It can write essays, summarize documents, hold conversations, and produce many valid responses to the same request. For most requests there is rarely a single correct string to compare against.

The diagram shows the core problem. Each valid response may be different from the others, and a reference string captures only one of them. An exact string comparison would mark a perfectly acceptable response as wrong simply because it used different words. This is why the simple picture of a test with one right answer breaks down for foundation models, and why evaluation needs methods that can judge responses that were never written down in advance.

The source points to an imbalance in effort. Investment in evaluation still lags behind investment in building models and applications, even though many new evaluation techniques are being developed. The gap is visible in everyday practice. A 2023 study by a16z found that 6 out of 70 decision makers evaluated models by word of mouth, a practice sometimes called a vibe check.

The emphasis on evaluation has also been noticed outside research papers. In December 2023, OpenAI cofounder Greg Brockman wrote that evals are surprisingly often all you need. When Terence Tao compared working with GPT-o1 to working with a mediocre but not completely incompetent graduate student, many people joked that qualified evaluators would soon be scarce, since judging future models could require the brightest human minds.

Many teams respond to the difficulty by turning toward human evaluation. The source treats humans in the loop as useful for sanity checks, and in many cases essential. Its own focus, however, is on automatic approaches, because those are the approaches that can be repeated and scaled, and because they lead directly to the question of how far an automatic score can be trusted.

The source groups the approaches to evaluating open-ended responses into three families: functional correctness, similarity scores, and AI as a judge. The first two are exact. The third is subjective. This split is the key to the rest of the topic, because the two families behave differently when you try to trust their numbers. Exact approaches produce scores that come from a fixed rule applied to the output. Subjective approaches produce scores that depend on a judgment, and that judgment is shaped by the judge that made it.

The table makes the contrast concrete. Exact approaches are predictable because the same rule gives the same result. AI as a judge is attractive because it can assess open-ended responses that exact checks cannot handle, but that flexibility is exactly what makes its score harder to interpret. The source is explicit that the two families should not be read in the same way.

To see why an AI judge's score needs care, follow one response from the moment it is generated to the moment a number is reported. The steps below trace the path the source describes. Each step adds a condition that the final number depends on.

AI as a judge is promising, but the source is careful about its weaknesses. Two limits matter most: a judge's score depends on the judge that produced it, and a judge's score can drift over time. Together they explain why the source concludes that AI judges should be supplemented with exact evaluation, human evaluation, or both.

There are two broad ways to compare several models. You can evaluate each model independently and then rank them by their scores. Or you can rank them using comparative signals, asking a direct question about a pair: which of the two models is better? The second approach is comparative evaluation. It is common in sports, especially chess, and it is gaining traction in AI evaluation.

Comparative evaluation depends on preference signals, and so does the post-training alignment process. These signals are expensive to collect. That cost motivated the development of preference models, which are specialized AI judges that predict which response users prefer. Preference models are used both in comparative evaluation and in post-training alignment. Since they are AI judges, they inherit the caution from the previous section: their predictions are only as trustworthy as the process that built and checks them.

Put the pieces together and the evaluation process becomes a set of layers rather than a single score. Exact checks give results that a fixed rule can reproduce. AI judges extend coverage to open-ended responses, but their scores need supplementing. Human review corrects and sanity-checks the judge. Comparative ranking then orders models by pairwise signals. The diagram below shows one way to arrange these layers. The order is a design sketch for illustration, not a sequence the source prescribes.