Testing Programs Against Valid and Invalid Input

A common first instinct is to judge a program by how it behaves when things go wrong. If the program crashes, the reasoning goes, then it must be incorrect. If it stays calm and produces a neat result every time, it must be correct. This instinct feels reasonable, but it merges two separate questions: whether a program does its job, and whether it survives situations it was not meant to face. This article separates those two questions. The two ideas it keeps apart are called correctness and robustness. Both are ways of evaluating a program, but they measure different things.

Before reading on, commit to an answer to the following scenario. Imagine a program whose job is to read a list of numbers from the user and print them in sorted order. Suppose that for every valid list of numbers it sorts correctly, but when the user types a word such as seven instead of a number, the program stops with an error and quits. Would you call this program correct? Would you call it robust? Decide on both answers before continuing.

The sorting program from the opening scenario has a single designed task: read numbers and print them in sorted order. Follow what happens to its flow under two kinds of input. A valid input is a list of numbers the program is meant to handle. An illegal or unexpected input is something outside that expected range, such as a word where a number was expected. The diagram below traces the two paths. Notice that they diverge at the moment the program checks the input, and that the two paths lead to different kinds of outcome.

The standard for correctness is the task itself. For the sorting program, correctness means producing sorted output for any set of input numbers it is meant to accept. Correctness is tested with the inputs the program is supposed to handle, and it asks a single question: does the program reach its goal? A single set of numbers that comes out unsorted is enough to show that the program fails its task.

Robustness is tested with inputs outside the expected range, and it asks whether the program fails gracefully. The standard here is reasonableness, not perfection. The sorting program does not need to guess what the user meant by a word, and it does not need to recover every possible value. It needs to avoid a crash or nonsensical output, and to respond in a way a sensible person would consider acceptable.

The source contrasts this with a non-robust program. Given the same bad input, such a program might crash or give nonsensical output. Both programs could be identical on every valid list of numbers. The difference appears only when the input falls outside what the program was built to expect.

Correctness and robustness are separate properties, so a program can be correct without being robust. The clearest way to see this is to put the two standards side by side. Each property has its own test inputs, its own question, and its own example from the sorting program. The comparison below shows how the same program is judged under each standard.

Because the conditions differ, the two verdicts can diverge. A program that sorts every valid list correctly but crashes on the word seven is correct and not robust. A program that politely reports an error for every bad input but sorts the valid lists incorrectly is robust and not correct. The first of these is the scenario the article opened with. Its crash tells you about robustness, and by itself says little about whether the program does its job.

The source makes an important distinction. Every program should be correct, but not every program needs to be completely robust. Whether it does depends on who will use the program and how it will be used. The same program can reasonably need different levels of robustness depending on its audience.

When deciding how much robustness to build, ask two questions drawn from the source: who will use the program, and how will they use it? A private tool used only by its author needs less invalid-input handling than a public program that strangers will feed unpredictable data. Correctness, however, is required in both cases.