Benchmarking Foundation Models

Imagine two teams announce that their model scored highly on the same test. One headline says the model is the best available. The other says it is only average. Both numbers may be accurate. The difference often lies in how each model was run, what it was asked to do, and how the result was measured.

This article is about the step that makes such numbers meaningful: evaluation during model adaptation. Adapting a model means changing it, or changing how it is used, so that it fits a particular application. Evaluation is the process of checking whether that adapted model is working as expected. Keep in mind that the source frames evaluation with two purposes: mitigating risks and uncovering opportunities.

A common misunderstanding is that evaluation is a final exam given once a project is finished. The source states that evaluation is needed throughout the whole adaptation process and names four jobs it performs. The diagram below shows those four stages in the order the source lists them.

The first stage is selecting which model to use. The second is benchmarking progress, which lets you tell whether changes made during adaptation are moving the system in the right direction. The third is determining whether an application is ready for deployment. The fourth begins once the application is running in production, when evaluation helps detect issues and find opportunities for improvement.

Each stage catches a different kind of risk. A model chosen badly is caught at selection. An adaptation change that makes things worse is caught while benchmarking. An application that looks good in a lab but is not ready is caught before deployment. A problem that only appears with real users is caught in production monitoring. Waiting for a single final evaluation would miss most of these.

To see why evaluating foundation models is harder, start with a simpler kind of task. In a closed-ended machine learning task such as fraud detection, there are usually expected answers, called ground truths. You compare the model's output against the expected answer. If they differ, you know the model is wrong.

Now consider a chatbot. For each prompt there are so many acceptable responses that nobody can write down a complete list of correct answers to check against. A response that differs from a stored answer is not necessarily wrong, and a response that matches one is not the only good option. The comparison step that worked neatly for fraud detection stops being a reliable check.

The source gives two reasons evaluation matters even more for foundation models. The first is these open-ended outputs. The second is the large number of adaptation techniques available. Because a free-form answer cannot be checked against a fixed list, evaluation has to judge outputs in other ways, and the way it is done affects the result.

The second reason evaluation is harder is that many adaptation techniques exist, and a system that performs poorly with one technique may perform much better with another. A prompt engineering technique is a method for shaping the input given to a model so that it performs better. Prompt engineering techniques and the number of examples provided can change which model appears to perform better.

This means a benchmark score is not a property of the model alone. It is a property of the model together with the setup that produced it. Changing the prompt wording, the examples shown, or the output format can change the number, even when the model itself has not changed. That is why a benchmark score should always be read together with its setup.

A benchmark comparison table only tells you something useful when the rows being compared were produced the same way. The source's central point is that a score should always be read together with the setup that produced it. The table below lists the parts of the setup that must match before two scores can be compared directly, and what goes wrong when they do not.