Designing Test Cases for Generated Code

When a model generates code, it is tempting to ask whether the code looks reasonable. Clean indentation, sensible variable names, and a plausible structure all feel like signs of success. The source is clear that this is not the first question to ask. The first question is whether the code does what it was supposed to do.

The same idea applies beyond code. If you ask a model to build a website, you check whether the website meets your requirements. If you ask it to make a restaurant reservation, you check whether the reservation succeeds. In each case the evaluation is based on behavior, not appearance. This article focuses on the code case, because there functional correctness can be measured automatically.

Functional correctness is not a new idea created for AI. Long before AI was used to write code, automatically verifying functional correctness was standard practice in software engineering. Coding platforms such as LeetCode and HackerRank use functional correctness evaluation to validate submitted solutions. A submission is accepted or rejected based on whether it produces the right results, not on how it reads.

A measurement of functional correctness has two stages, and they answer two different questions. The generated code is first passed to a Python interpreter to check whether it is valid. Only if it is valid does the second question arise: for a given input pair, does the code output the correct result?

Consider a checking program that tests a generated gcd function against a list of input pairs. Each pair comes with an expected greatest common divisor. The program prints one line per pair and labels each mismatch with the expected value and the value the function actually produced. Before you read the output, think about what the report will contain.

Read the report line by line. Each line beginning with FAIL names one input pair, then states the expected value and the value the function produced. The line FAIL (15, 20): expected 5, got 15 is the mismatch the source describes. The correct greatest common divisor of 15 and 20 is 5, so a function that returns anything else for that pair is wrong. The report shows exactly where the function diverged from the expected result, which is the information a developer needs in order to debug it.

Follow one test through the flowchart. The test case supplies an input pair and an expected value. The generated function receives the input and returns an actual value. The comparison step then sends the flow to one of two terminal results. The same flow repeats for every test case in the set. This is why a single function can produce several FAIL lines in one report.

A unit test executes code in a scenario and compares the actual output to the expected output. A test case is therefore a scenario plus its expected output. The scenario is the input the code receives, and the expected output is the value that code must return for that input. Neither part is useful alone: a scenario without an expected output has nothing to check against, and an expected output without a scenario does not say what was being tested.

A single test case tells you about one input. Code is typically validated with unit tests, in which the code is executed in different scenarios to ensure that it generates the expected outputs. Each scenario is one test, and the whole set of scenarios is what gives a measurement of functional correctness its coverage. A function that passes one test but fails many others is not functionally correct for the intended behavior.

Benchmarks are standardized datasets and metrics used to evaluate AI models on code generation tasks. A benchmark problem comes with a set of test cases, and the generated output is checked against those test cases. The source gives an example from HumanEval, a benchmark from OpenAI: a problem asks for a function named has_close_elements, which takes a list of floating-point numbers and a threshold. The problem description includes example calls, and each test case pairs one scenario with the output the function should return for it. The test cases are what the generated function is checked against.

The same measurement idea extends beyond Python. Spider, BIRD-SQL, and WikiSQL are benchmarks for text-to-SQL, where the generated output is a SQL query. Their test cases decide whether the generated output is accepted, just as the Python benchmarks' test cases do. What changes between benchmarks is the kind of output produced, not the principle of checking that output against expected results.