Evaluating Code Generation Benchmarks

Suppose a model hands you a function that looks clean, well named, and sensible. It is tempting to conclude that the code is good. But when a model generates code, the first question is not whether the code looks reasonable. The question is whether it does what it was supposed to do. Answering that question requires a different kind of check than reading the code, and for code generation that check can often be performed automatically.

This article focuses on the code case, where functional correctness can be measured automatically. We will start with a concrete run of a checking program so you can see what a functional correctness report looks like, then build up the ideas behind it: what a test case is, how a harness decides between pass and fail, and how public benchmarks use the same idea to score AI systems.

Functional correctness evaluation means evaluating a system based on whether it performs the intended functionality. The idea is not special to code. The same question applies to any system you ask a model to build or operate, and the check is always about the outcome the system is supposed to produce.

Coding platforms apply the same logic. Functional correctness evaluation is how coding platforms like LeetCode and HackerRank validate submitted solutions, and long before AI was used to write code, automatically verifying functional correctness was standard practice in software engineering.

Consider a program that checks a generated gcd function against a list of input pairs, where each pair has an expected greatest common divisor. The program prints one result per pair, and it labels each mismatch with the expected value and the value the function actually produced. The function below is a deliberately flawed candidate, written for this lesson, that returns the smaller of its two inputs instead of computing the greatest common divisor. The harness runs it on four pairs.

Read the report one line at a time. Each line beginning with FAIL names one input pair, then states the expected value and the value the function produced. A line such as FAIL (15, 20): expected 5, got 15 is a mismatch: the correct greatest common divisor of 15 and 20 is 5, so a function that returns anything other than 5 for that pair is wrong. The report shows exactly where the function diverged from the expected result, which is the information a developer needs in order to debug it.

The PASS line in the report deserves attention too. The pair (8, 4) passes only because the smaller input happens to be the greatest common divisor for that pair. One matching line is not evidence that the function is correct. A single successful check tells you about that one scenario, and nothing more.

A measurement of functional correctness has two stages. First, the generated code is input into a Python interpreter to check whether it is valid. If the code is not valid, it cannot be run at all. If it is valid, the next question is whether it outputs the correct result for a given input pair.

In the candidate_gcd example, every line of the program runs without error. The FAIL lines are not crashes. They are correctness failures, and only the comparison step reveals them.

The diagram below traces one test. The test case supplies an input pair and the expected value. The generated function receives the input and returns an actual value. The comparison step then decides between the two terminal results: PASS when the actual value matches the expected value, and FAIL when it does not. The same flow repeats for every test case in the set, which is why a single function can produce several FAIL lines in one report.

Trace the diagram with the (15, 20) pair. The test case supplies the inputs 15 and 20 with expected value 5. The function returns 15. The comparison finds that 15 is different from 5, so the path leads to FAIL. The harness then moves to the next test in the set, and the loop continues until every case has been checked.

A test case is a scenario plus its expected output. The scenario is the situation the code is executed in: the inputs it receives. The expected output is what the code should return for that scenario. A unit test executes code in a scenario and compares the actual output to the expected output. Each scenario is one test, and the whole set of scenarios gives the measurement its coverage.

In the tuple (12, 8, 4) from the harness, the first two values form the scenario and the last value is the expected output. Reading the case this way makes the mapping explicit: the function is run on the scenario (12, 8), and its actual return value is checked against the expected output 4.

Code is typically validated with unit tests, in which the code is executed in different scenarios to ensure that it generates the expected outputs. Unit testing is a standard practice in software engineering. Each scenario is one test, and the whole set of scenarios is what gives a measurement of functional correctness its coverage.

Coverage therefore depends on the test set. A function checked on only one pair has been measured in one scenario. The four pairs in the harness are a small set, and the flawed candidate was caught by three of them. A larger and more varied set of scenarios gives a broader measurement, while a narrow set can let a wrong function pass.

It helps to contrast functional correctness with judging code by its text or structure. A surface-level judgment asks whether the code looks like an acceptable answer. A functional judgment asks whether running the code produces the right results. The table below sets the two side by side.

A benchmark is a standardized dataset and metric used to evaluate AI models on a task. For code generation, a benchmark problem comes with a set of test cases, and the generated code is checked against them. The pipeline below shows the path a problem takes from prompt to final score.