Benchmarks and Saturation in Model Evaluation

Most people who have trained a small predictive model learn to judge it with a simple routine. You prepare inputs whose correct answers are already known, run the model on them, and count how often its output matches the expected answer. This routine works well when the model is choosing from a fixed menu of possible results. Foundation models, however, break the routine in several ways at once. This article follows three of those breaks and then looks at a fourth change in what evaluation is supposed to accomplish.

Follow one pass through the familiar routine step by step. First, the evaluator prepares a set of inputs and attaches to each one a correct answer. Second, the model receives each input and produces an output. Third, the evaluator compares each output with its expected answer and counts the matches. The final score is simply the share of matches. Every step depends on one assumption: that there is a known expected answer for each input that the evaluator can compare against.

Keep this assumption in mind as the article proceeds. Each of the four changes that follow is a way in which the assumption stops holding for foundation models.

Open-ended tasks change the structure of that check. For a single input, there can be a large number of responses that are all correct. A request to draft a sympathetic reply to a customer, or to explain a scientific idea to a child, has no single right string of words. Because of this, it is impossible to build a complete list of acceptable outputs and compare the model against it.

The diagram shows the core problem. In the fixed case, one expected string is compared against the output, and the result is a clear match or miss. In the open-ended case, several correct replies exist, but an answer key that lists only some of them compares every output against an incomplete set. Valid replies that were never listed get marked as misses, and the key gives no information about the responses it never covered.

Many foundation models are black box models. They hide their architecture, their training data, and their training process. From the evaluator's side, the model is a box with inputs going in and outputs coming out. The diagram below shows what can be seen and what remains hidden.

Only the inputs and outputs are visible. The hidden details are not trivial. The architecture, the training data, and the training process can each reveal where a model is likely to be strong and where it is likely to fail. When those details are hidden, the evaluator has only one source of evidence: the outputs produced in response to inputs. Testing becomes an exercise in watching behavior from outside, and any explanation of why the behavior occurs must be inferred rather than read directly.

A benchmark is saturated for a model once the model achieves the perfect score. Foundation models saturate benchmarks quickly, and this forces repeated replacement. The timeline below lists the benchmarks named in the source material, in the order they were introduced or replaced.

Read the timeline as a sequence of ceilings. Each benchmark was introduced as a way to tell models apart. Once models began reaching the perfect score, the benchmark stopped separating a better model from a worse one, and a replacement was needed. GLUE became saturated within about a year of its 2018 introduction, which led to SuperGLUE in 2019. NaturalInstructions in 2021 followed a similar pattern and was succeeded by Super-NaturalInstructions in 2022. MMLU, introduced in 2020, was a strong benchmark that many early foundation models relied on, and it was largely replaced by MMLU-Pro in 2024.

The fourth change concerns the purpose of evaluation itself. Evaluating general-purpose models means both measuring known tasks and discovering new tasks, and the discovery work may include tasks that go beyond human capability. An evaluator is not only checking a list of expected abilities. The evaluator is also trying to find abilities nobody wrote down in advance and boundaries nobody has yet mapped.

This is why a fixed benchmark, built around known tasks, cannot be the entire evaluation effort for a general-purpose model. A benchmark can only score what its authors thought to include. Saturation adds pressure here: once known tasks are solved, the evaluator must look for new ones to learn anything further.

The field has responded by building new evaluation methods and new benchmarks. The source material reports that the number of published papers on LLM evaluation grew exponentially during the first half of 2023, rising from about 2 papers per month to almost 35 papers per month. That growth reflects how seriously the community treats the problem of measuring what foundation models can do.