Evaluating Foundation Models: Challenges and Limitations
Evaluating ML models is already difficult, and foundation models make evaluation even harder.
Two Students and One Hard Question
Evaluating a machine learning model has always been a difficult job. Once foundation models arrived, the job became harder still. This article explains why, and it starts with a thought experiment before any theory.
What do you think happens?
Two students each hand you a math solution, and you must decide whether it is correct. One is a first grader and the other is a PhD candidate working on a hard problem. For which solution would you feel more confident making a judgment?
Reveal answer
The first grader's solution. You would feel more confident judging it.
A first grader's problem is simple enough that you can check each step yourself. The PhD candidate's solution addresses a hard problem, so verifying it requires more expertise and effort than you may have on hand.
Flaws on the Surface and Flaws Underneath
The same pattern appears outside of mathematics. Suppose a model writes a summary of a book. If the summary is gibberish, you can reject it almost immediately, because its problems are visible on the surface. Nothing about the judgment requires deep knowledge of the book. The flaw announces itself.
A coherent summary is different. Reading it tells you little about whether it is accurate. A smooth, well-organized summary can still misrepresent the book, and the only reliable way to check it may be to read the book yourself first. The quality that makes the summary pleasant to read is the same quality that hides its errors.
Why Smarter Models Are Harder to Check
In this context, intelligence means the model producing output that is harder for an evaluator to judge. A weak response often shows its flaws openly. A strong response may look plausible while hiding subtle errors.
The central idea is simple to state: the more intelligent AI models become, the harder it is to evaluate them. This is because the difficulty of evaluation is not a fixed property of a task. It grows alongside the sophistication of what the model produces. A judge who can easily assess simple output may be unable to assess advanced output without extra expertise or effort.
Foundation models sit on top of a problem that already existed. Evaluating a machine learning model was already difficult before foundation models arrived. The newer models add to that difficulty because their outputs are more sophisticated, so the gap between what an evaluator can easily see and what the model actually did grows wider.
Why Sophisticated Judgments Take Longer
A second consequence follows from the first. Evaluation can be far more time-consuming for sophisticated tasks. When a response is easy to judge, a quick look may be enough. When a response is sophisticated, the evaluator may need to gather background material, work through the reasoning step by step, or consult a reference such as the original book. Each of these steps adds time.
Because of this, evaluators should expect and plan for the extra effort. A review schedule that allots the same few minutes to every output will be too short for sophisticated tasks, even though it may be fine for simple ones.
Mistakes That Produce Unreliable Judgments
Common mistake Judging a fluent output by its polish alone
What it looks likeA coherent, well-organized summary of a book is accepted because it reads smoothly.
Why it failsA smooth summary can still misrepresent the book. Readability says little about accuracy.
Do this insteadCheck claims against the source material, which may mean reading the original book.
Common mistake Using a simple-task check on a sophisticated output
What it looks likeAn evaluator looks only for obvious nonsense, finds none, and approves the output.
Why it failsSimple checks catch surface flaws. Sophisticated output may have subtle errors that such a check cannot see.
Do this insteadMatch the depth of the check to the sophistication of the output, adding expertise or effort where needed.
Common mistake Assuming an advanced answer is correct because it sounds plausible
Why it failsThe source notes that a strong response may look plausible while hiding subtle errors.
Do this insteadWork through the reasoning step by step before deciding.
Common mistake Planning a review with the same time for every output
Why it failsSophisticated tasks require gathering background, step-by-step reasoning, and consulting references, and each step adds time.
Do this insteadExpect and schedule extra time for sophisticated tasks.
Practice Judging a Summary
A model produces two summaries of the same book. Summary A is a jumble of unrelated sentences. Summary B is clear, well organized, and reads convincingly. Which summary can you reject quickly, which one requires more work to judge, and what would you need to do before you could trust Summary B?
Hints
- Ask which flaws are visible on the surface and which are not.
- Think about what a reliable check of an accurate-looking summary would require.
At a glance
- Evaluating ML models is already difficult, and foundation models make evaluation even harder.
- The more intelligent a model becomes, the harder its output is to judge, because flaws in sophisticated output are less visible.
- Simple output with visible flaws can be rejected quickly, while a coherent output may require background reading, step-by-step reasoning, or consulting the original source to verify.
- Sophisticated evaluations take longer, and common mistakes include judging by polish, applying simple checks to advanced output, and under-planning the time required.