Evaluating models with benchmarks and evaluation harnesses

Suppose you are choosing a model for a product. A tempting shortcut is to look for the model with the highest benchmark score and use it. That shortcut hides several questions. What did the benchmark actually test? How was the model run against it? Does the top score matter for your application at all? Evaluating a model well means understanding the tools that produce scores, not just reading them.

This article follows one thread from start to finish. It begins with how benchmarks are run and scored, then looks at how open source and proprietary models compare, and finally turns to the practical trade-offs of cost, supporting features, control, and provider risk. Keep one question in mind throughout: what would your application need to keep working in a year, and which model choice makes that easier to guarantee?

A benchmark is a set of test cases built to measure some ability. To run many benchmarks at once, people use an evaluation harness, which is a tool that evaluates a model across multiple benchmarks. The harness handles the repetitive work so that every model is tested the same way. Follow one test case through the process to see where the model's answer is actually judged.

The important point is the order of steps. The model never judges its own answer. The harness sends the test prompt, keeps the output, and only then applies a scoring step. The judgment happens after the model has finished responding, which is why two harnesses that score differently can report different numbers for the same model even when the model's raw answers are identical.

There are thousands of benchmarks, each designed to measure a different model capability. Google's BIG-bench, published in 2022, contains 214 benchmarks on its own, and the number keeps growing as AI use cases multiply. The benchmarks cover a wide range of abilities, from math and puzzle solving to recognizing words written as ASCII art.

Benchmarks also wear out. As models improve, an older benchmark stops separating strong models from weak ones, because most models score near the top. New benchmarks then have to be introduced so that evaluation keeps measuring something useful. A high score on an old benchmark therefore tells you less than it once did.

Two harnesses are named in the source. EleutherAI's lm-evaluation-harness supported over 400 benchmarks at the time of writing. OpenAI's evals lets you run roughly 500 existing benchmarks and register new ones for OpenAI models. Both give you a structured way to run a model through many tests, which is the main purpose of a harness.

A common assumption is that open source models will eventually match the best proprietary models. The evidence points in a more nuanced direction. Across published benchmarks, the distance between open source and proprietary models has been shrinking. The benchmark MMLU is often used to illustrate this. If you plotted scores for each group against time, you would see two curves moving toward each other. The proprietary curve stays higher, but the space between the curves narrows with each new generation.

A narrowing gap does not mean the gap is gone. The source argues that incentives explain why. A developer who holds the strongest model available has a strong reason to keep it behind an API, where it can be monetized, rather than releasing it for others to use. Companies often keep their strongest models proprietary while open sourcing weaker ones. The likely result is that the strongest open source model will keep trailing the strongest proprietary models for the foreseeable future.

There is a second, quieter reason for the lag, and it is separate from raw model quality. Commercial providers learn how their models are used in practice, through the feedback and behavior of their users. Once an open source model is released, its developers usually have no direct view of how it is used or how well it holds up in the real world. That missing feedback loop is a structural disadvantage.

A model on its own rarely serves a use case. Several supporting functions have to exist around it before users can rely on the result. The source names scalability, function calling, structured outputs, and output guardrails as examples. Each answers a different production question, and each is a place where the deployment choice matters. Logprobs, which expose the model's probability estimates for its outputs, are another such function, and they are valuable for classification, evaluation, and interpretability.

Building these functions yourself is slow and demanding, which is one reason many teams choose a commercial API. Providers often ship some of them ready to use. The trade-off is that you can only use what the provider chooses to offer. Commercial providers may hesitate to expose logprobs, partly because others could use them to imitate the model. In practice, many APIs do not expose logprobs at all, or expose only a limited amount. If your design depends on that signal, a commercial API may simply not give it to you.

The cost comparison is less intuitive than it first appears. Using a model API means your spending scales with your usage, and at heavy volume that spending can become very large. Self-hosting trades that recurring charge for a different kind of expense. You take on the time, talent, and engineering effort needed to optimize the model, keep the inference service running at the right scale, and build the guardrails that protect your product. The source sums up the tension this way: APIs are expensive, but engineering can be even more so.

The reliability piece matters too. Using another company's API means your product depends on that company's service-level agreement. If the API is unreliable, which the source notes is often true for early-stage startups, you will spend engineering effort on guardrails around the provider's shortcomings. So the cost of an API is not only the invoice. It can also include engineering work you did not plan for.

When a business depends on a model, it is reasonable to want real influence over how that model behaves. A 2024 study by a16z, cited in the source, found that enterprises care about open source models mainly for two reasons: control and customizability. Control and customizability are the two things that deployment choices most directly affect. Whether you can finetune or freeze a model matters when a business relies on it for the long run.

A good model choice is one that is easy to use and to manipulate. The source observes a practical split between the two families. Proprietary models are generally easier to get started with and scale, while open models can be easier to manipulate. Whichever family you pick, the source recommends that the model follow a standard API, because a standard interface makes it easier to swap one model for another. The source also suggests that deployment constraints significantly narrow your options before you compare performance numbers.

Some applications must run on the device itself, and in that case third-party APIs are simply not an option. Local operation is desirable when the application must work where internet access is unreliable, or when privacy matters, such as an assistant that can see all of a user's data but should not send that data off the device. Self-hosting can make on-device use possible, although the source notes it may be hard to do well.