Reference-Based and Reference-Free Metrics
The metric this article begins with, pass@k, is built on that idea of checking real behavior rather than comparing text.
Judging Code by Whether It Works
Suppose you have a model that writes code and you want to know how good it is. One option is to read each program and judge it by eye. That is slow, and two reviewers may disagree about the same program. A more reliable question is whether the program actually does its job. Each programming problem comes with test cases, and a solution either passes them or it does not. The metric this article begins with, pass@k, is built on that idea of checking real behavior rather than comparing text.
What do you think happens?
A model generates three programs for the same problem. The first one fails the test cases, the second one passes all of them, and the third is never checked. Is the problem counted as solved?
Reveal answer
Yes
A problem counts as solved if any of the k generated samples passes all of its test cases. The second program passes everything, so the problem is solved even though the first sample failed.
What Functional Correctness Measures
Functional correctness evaluates AI outputs by measuring whether they achieve a measurable objective, such as a game score or energy savings.
The idea behind functional correctness is that the output is judged by what it does, not by how it looks. A game-playing agent can be judged by the score it reaches. A building-control system can be judged by the energy it saves. In code generation, the measurable objective is passing a problem's test cases. The common thread is a measurable objective, and tasks that have one can usually be evaluated this way.
Before you use functional correctness for an AI system, ask whether the task has a measurable objective that can be checked automatically. If it does, the output can be judged by its behavior. If it does not, the evaluation needs another approach.
Following One Problem Through Its Samples
The definition is easier to hold onto if you follow a single problem through its k samples. Picture one problem with a set of test cases. The model writes a first program, and the test cases run against it. If that program fails, the model writes a second program for the same problem, and the test cases run again. The problem is counted as solved the moment any one of these programs passes everything. The diagram below shows this sequence, with the samples arriving one after another.
This diagram shows one possible path, where the second sample is the one that passes. The same counting rule applies whichever sample succeeds. What matters for pass@k is only whether a passing sample exists among the k generated for that problem. A first sample that passes would end the sequence right away, and a problem where no sample passes stays unsolved.
Counting Solved Problems Across a Set
The score counts problems, not individual samples. A problem where only one of the k samples works still counts as solved. A problem where none of the k samples works counts as unsolved, no matter how close the failing attempts came. This is why the metric is read as a problem-level success rate. With S solved problems out of N total problems, the calculation is a simple fraction.
The diagram below shows the same calculation for a whole problem set. Each problem first receives a solved or unsolved result from its own samples. Those results are then counted and divided by the total number of problems.
Why pass@k Rises as k Grows
Because a problem is solved when any of its k samples passes, adding more samples gives the model more opportunities to succeed. Each extra sample can only help the chance of solving that problem, because a problem that was already solved stays solved and an unsolved problem may become solved. This is the reason the score tends to rise as k grows.
What do you think happens?
For the same set of problems and the same model, will pass@3 be lower than, equal to, or higher than pass@1?
Reveal answer
Equal or higher
Adding samples can only add chances to solve a problem. A problem solved by its first sample is still solved when two more samples are allowed, so pass@3 can never be lower than pass@1 on the same problems.
Calculating pass@k in a Worked Example
Scoring four problems with three samples each
A model is given four problems and generates three samples for each one. Each sample is marked pass or fail by running that problem's test cases. Compute pass@3 for the set. Then compute pass@1 using only each problem's first sample, and explain the difference.
- 1Read problem 1
Samples are fail, pass, fail. The second sample passes, so problem 1 is solved at k = 3. Its first sample alone fails.
- 2Read problem 2
Samples are fail, fail, fail. No sample passes, so problem 2 is unsolved even though it was attempted three times.
- 3Read problem 3
Samples are pass, fail, fail. The first sample passes, so problem 3 is solved at both k = 1 and k = 3.
- 4Read problem 4
Samples are fail, fail, fail. Problem 4 is unsolved.
- 5Count solved problems at k = 3
Problems 1 and 3 are solved, so S = 2 and N = 4. Then pass@3 = 2 / 4 = 0.5.
- 6Count solved problems at k = 1
Only the first sample of each problem is used. Only problem 3's first sample passes, so S = 1 and pass@1 = 1 / 4 = 0.25.
- 7Explain the difference
Problem 1 was missed at k = 1 only because its passing program was the second sample. Allowing more samples raised the score from 0.25 to 0.5 on the same problems.
pass@3 = 0.5 and pass@1 = 0.25 for this set. The score rose as k increased, and the two values are only comparable as reported at their own k.
| Problem | Sample results in order | Solved at k = 1? | Solved at k = 3? |
|---|---|---|---|
| Problem 1 | fail, pass, fail | No | Yes |
| Problem 2 | fail, fail, fail | No | No |
| Problem 3 | pass, fail, fail | Yes | Yes |
| Problem 4 | fail, fail, fail | No | No |
| Total solved | 1 of 4 | 2 of 4 |
The same four problems scored at two different values of k
Reference-Based Metrics and Reference Data
Not every task can be checked by running a program or reading a game score. When the task you care about cannot be evaluated through functional correctness, a common alternative is to compare the model's output against reference data. The source names the two families here. Metrics that need reference data are called reference-based. Metrics that do not need it are called reference-free. Functional correctness and pass@k belong to the group that checks outputs by their behavior, while reference-based metrics depend on having a set of correct answers to compare against.
Choosing Between the Two Metric Families
The two families differ in what the output is compared against. Functional correctness checks the output's behavior against a measurable objective, such as passing test cases. A reference-based metric checks the output against stored correct responses. The diagram below sets the two side by side, showing the inputs each one needs.
Use functional correctness when the task has a measurable objective that can be checked automatically. Use a reference-based metric when that kind of check is not possible and you have reference data you trust.
Common Mistakes When Reading These Scores
Common mistake Counting individual samples instead of problems
What it looks likeA problem with samples fail, pass, fail is treated as one passing sample and two failing samples.
Why it failspass@k is a problem-level success rate, so the problem counts once as solved.
Do this insteadCount the problem as solved if any of its k samples passes, and divide the number of solved problems by the total number of problems.
Common mistake Giving partial credit for close failing attempts
What it looks likeA problem where all three samples fail but came close is counted as half solved.
Why it failsA problem where none of the k samples works counts as unsolved, however close the attempts came.
Do this insteadScore the problem as unsolved and keep the closeness of the attempts out of the pass@k calculation.
Common mistake Comparing scores reported at different k values
What it looks likeA model reported at pass@10 is ranked above another model reported at pass@1.
Why it failspass@10 gives ten attempts per problem and pass@1 gives one, so the two numbers measure different things.
Do this insteadCompare models only at the same k.
Common mistake Assuming reference data is automatically correct
What it looks likeA reference-based score is reported with no check of how the reference answers were produced.
Why it failsThe quality of reference data depends on how it was generated and reviewed.
Do this insteadCheck how the references were created and reviewed before trusting the comparison.
Practice: Scoring a New Problem Set
A model is given five problems and generates two samples for each one, so k = 2. Problem A: fail, pass. Problem B: fail, fail. Problem C: pass, pass. Problem D: fail, fail. Problem E: fail, pass. Compute pass@2 for this set. Then compute pass@1 using only each problem's first sample, and state which value is larger and why.
Hints
- A problem is solved if any of its k samples passes, so check each problem's two results in order.
- Count the solved problems first, then divide by the total number of problems, which is 5.
- For pass@1, look only at the first result of each problem and ignore the second.
Key Takeaways
- Functional correctness judges an AI output by whether it achieves a measurable objective, such as passing a problem's test cases, rather than by how the text looks.
- pass@k is the fraction of problems solved, where a problem counts as solved if any of its k generated samples passes all test cases, so it is read as a problem-level success rate.
- pass@k tends to rise as k increases because more samples give more chances to succeed, so scores should be compared only at the same k.
- When automatic checking is not possible, reference-based metrics compare outputs against reference data, and the trustworthiness of that comparison depends on how the references were generated and reviewed.