Challenges of Evaluating Foundation Models
Evaluation mitigates risks and uncovers opportunities across model selection, benchmarking, deployment readiness, and production monitoring.
Two Teams, One Test
Imagine two teams announce that their model scored highly on the same test. One headline says the model is the best available. The other says it is only average. Both numbers may be accurate. The difference often lies in how each model was run, what it was asked to do, and how the result was measured.
What do you think happens?
When two models get different scores on the same benchmark, what should you check before deciding which model is better?
Reveal answer
The setup that produced each score: how the model was run, what it was asked to do, and how the result was measured.
A benchmark score should always be read together with the setup that produced it. A higher number produced under a different setup does not automatically mean a better model.
This article is about the step that makes such numbers meaningful: evaluation during model adaptation. Adapting a model means changing it, or changing how it is used, so that it fits a particular application. Evaluation is the process of checking whether that adapted model is working as expected. The source frames evaluation with two purposes that you should keep in mind as you read: mitigating risks and uncovering opportunities. Evaluation is not only a final exam at the end of a project. It is a way of seeing what could go wrong and what could be made better.
Checking Against a Known Answer
To see why evaluation is harder for foundation models, start with a simpler kind of task. In a closed-ended machine learning task such as fraud detection, there are usually expected answers, called ground truths. You compare the model's output against the expected answer. If they differ, you know the model is wrong.
Now consider a chatbot. For each prompt, there are so many acceptable responses that nobody can write down a complete list of correct answers to check against. A response that differs from a stored answer is not necessarily wrong, and a response that matches one is not the only good option. The comparison step that worked neatly for fraud detection stops being a reliable check.
Evaluation Across the Adaptation Lifecycle
Evaluation is not a single event. The source states that it is needed throughout the whole adaptation process and names four jobs it performs. First, it helps select which model to use. Second, it benchmarks progress, so you can tell whether changes made during adaptation are moving the system in the right direction. Third, it determines whether an application is ready for deployment. Fourth, once the application is running in production, evaluation helps detect issues and find opportunities for improvement.
The source also notes that evaluation has always mattered in machine learning engineering, but that it matters even more with foundation models. It gives two reasons: the open-ended outputs described above, and the large number of adaptation techniques available. The second reason is the one that connects evaluation to the prompt setup discussed next.
How the Prompt Setup Changes the Score
A prompt engineering technique is a method for shaping the input given to a model so that it performs better. The number of adaptation techniques makes evaluation harder because a system that performs poorly with one technique may perform much better with another. The same model can therefore appear stronger or weaker depending on the setup used to run it.
The source gives a concrete case. When Google launched Gemini in December 2023, it claimed that Gemini was better than ChatGPT on the MMLU benchmark. The source explains that Google had evaluated Gemini using a prompt engineering technique called CoT@32, in which Gemini was shown 32 examples, while ChatGPT was shown only 5 examples. The two numbers were produced under different setups, so the headline comparison did not hold the conditions constant.
Reading a Benchmark Table Fairly
A benchmark comparison table places several models side by side. Before you compare their scores, identify the conditions that were used to produce each score. The source's central point is that a benchmark score should always be read together with the setup that produced it. A fair comparison therefore requires that the conditions behind each column match, not just that the models have the same name in the header.
| Condition to check | Why it must match | Question to ask of the table |
|---|---|---|
| Prompt engineering technique | A different technique can change which model appears to perform better | Did every model use the same technique? |
| Number of examples in the prompt | The source shows 32 examples and 5 examples producing different headline comparisons | Was each model given the same number of examples? |
| How the result was measured | A benchmark score is only meaningful alongside how it was obtained | Was the same measurement method used for every row? |
| What the model was asked to do | The task definition shapes the output being scored | Was every model given the same task wording and format? |
Conditions that must match before benchmark scores can be compared directly. The source does not provide the numeric cells of a specific table, so this checklist is drawn from the conditions the source names.
Common Mistakes When Comparing Scores
Common mistake Treating a benchmark score as a fixed property of the model
What it looks likeA headline says Model X is the best because it scored higher on a test.
Why it failsThe score depends on the setup that produced it, including the prompt technique and number of examples.
Do this insteadAsk what setup produced the score before comparing it with another score.
Common mistake Assuming an open-ended answer can be checked like a fraud label
What it looks likeMarking a chatbot response wrong because it differs from a stored sentence.
Why it failsFor open-ended tasks there are many acceptable responses, so a difference from one stored answer is not necessarily an error.
Do this insteadRecognize that open-ended evaluation needs checks beyond a single expected answer.
Common mistake Evaluating only at the end of a project
Why it failsThe source says evaluation is needed throughout adaptation, including model selection, benchmarking, deployment readiness, and production monitoring.
Do this insteadPlan evaluation at each stage so problems are caught before they reach the next stage.
Common mistake Ignoring how the result was measured when reading a table
Checking a Pair of Reported Scores
Two teams report scores for their models on the same benchmark. Team A reports a higher score. Before you decide that Team A's model is better, list the setup details you would need to confirm: the prompt technique, the number of examples, the task wording, and the measurement method. Which of these, if they differed between the two teams, would make the comparison unfair?
Hints
- Look back at the Gemini example: which single detail differed between the two reported setups?
- A fair comparison requires the conditions behind each score to match, not just the model names.
- Any detail that changes the input the model sees, or how its output is scored, can change the reported number.
Key Takeaways
- Evaluation is needed throughout model adaptation, not only at the end: it supports model selection, benchmarking, deployment readiness, and production monitoring.
- Foundation models are harder to evaluate than closed-ended tasks like fraud detection because open-ended outputs have no complete list of correct answers.
- The prompt setup, including the technique and the number of examples, can change which model appears to perform better, as the Gemini and ChatGPT example shows.
- A benchmark score must be read together with its setup; a fair comparison requires matching prompt technique, number of examples, task definition, and measurement method.