Comparing Model Performance Fairly

Suppose you have a model that writes code, and you want to know how good it is. One option is to read each program and judge it by eye. That is slow, and two reviewers may disagree about the same program. A more reliable question is whether the program actually does its job. Each programming problem comes with test cases, and a solution either passes them or it does not. The metric this article covers, pass@k, is built on that idea of checking real behavior rather than comparing text.

The underlying principle is broader than code. Functional correctness evaluates AI outputs by measuring whether they achieve a measurable objective, such as a game score or energy savings. The common thread is a measurable objective. Tasks that have one can usually be evaluated this way, and pass@k is the version of this idea that applies to code generation, where the measurable objective is passing a problem's test cases.

The definition is easier to hold onto if you follow a single problem through its k samples. Picture one problem with a set of test cases. The model writes a first program, and the test cases run against it. If that program fails, the model writes a second program for the same problem, and the test cases run again. The problem is counted as solved the moment any one of these programs passes everything.

This diagram shows one possible path, where the second sample is the one that passes. The same counting rule applies whichever sample succeeds. What matters for pass@k is only whether a passing sample exists among the k generated for that problem. If the first sample passes, the problem is solved and the later samples do not change that result.

Notice that the score counts problems, not individual samples. A problem where only one of the k samples works still counts as solved. A problem where none of the k samples works counts as unsolved, no matter how close the failing attempts came. This is why the metric is read as a problem-level success rate.

In this formula, N is the number of problems in the set and S is the number of those problems that have at least one passing sample among their k generated samples. The result is always a fraction between 0 and 1.

Because a problem is solved when any of its k samples passes, adding more samples gives the model more opportunities to succeed. Each extra sample can only help the chance of solving that problem. This is the reason the score tends to rise as k grows. Scores are therefore expected to rise as k increases, which is why they should be compared at the same k.

The diagram uses the same five problems as the worked example. Moving from k = 1 to k = 2 adds two newly solved problems, which is a gain of 0.4. Moving from k = 2 to k = 3 adds one more, a gain of 0.2. Each extra sample still helps, but the increase is smaller than the one before because the easiest-to-solve problems are often already counted. Problem D stays unsolved throughout, so no value of k can raise the score for it.

Not every task can be checked by running a program or reading a game score. When the task you care about cannot be evaluated through functional correctness, a common alternative is to compare the model's output against reference data. Each entry in reference data pairs an input with one or more reference responses. These references are also called ground truths or canonical responses.

Reference data is usually produced by people, and increasingly by AI systems. When humans create the references, the human answers become the standard that the model is measured against. In that setup, human performance is treated as the gold standard, and the model's performance is judged by how closely it matches it. The quality of any reference-based score therefore depends on how the reference data was generated and reviewed.

Human-written references are often expensive and slow to produce, which is the main limitation of this approach. The amount of reference data you can build is limited by how much and how fast it can be generated. This pressure is why many teams turn to AI to generate reference data. AI-generated references may still need human review, but reviewing them takes far less effort than writing every reference from scratch.