Debugging Test Failures

Many programs are checked by small tests. A test runs a function on known input and looks for anything unexpected in what comes back. The test in this article checks find_all(), a function that searches a string for every occurrence of one character. Before reading any explanation of how it works, read the test itself and notice what it does with each result it receives.

The test contains no list of expected output and no counting. It makes one promise: every position that find_all() reports should hold the letter 'a'. If that promise is broken even once, a message appears. The test is therefore about the quality of each individual result, not about how many results there are.

To see the test succeed in detail, trace the string by hand. Positions count from zero, which is how positions in a C++ string are numbered. The letter M sits at position 0, the letter a at position 1, and so on through the whole sentence. The letter 'a' appears at four positions in this text: 1, 6, 9, and 19. These are the matches find_all() should report, and each one should pass the test's check.

The test loops over every match that find_all() returns. Inside the loop, a variable named p refers to one match at a time. The check inside the loop compares the character that p refers to with the character 'a'. The expression that reads that character is written *p, and it means the value stored at the place p refers to. The check is written in this form:

Comparing the value that p refers to, rather than p itself, is what makes the test examine the characters and not the positions. The iterator is the thing that moves from match to match. The value is what the rule is about. Because the test reads the value, it can check whether each match holds the right letter, without caring where that match sits in the string.

Now suppose find_all() were broken and reported a position that holds a different letter, such as the r at position 2. The loop would reach that result, the check *p equal to 'a' would fail, and the message a bug! would appear. The output alone tells you that a mismatch happened, but not which result caused it. The divergence is at the single loop iteration where the character under p is not 'a'.

A practical way to pin the failure down is to print the position or the character inside the loop as well. This turns a single error message into a record of every match that was checked. Once you can see that sequence, the first unexpected entry marks where the result diverged from what you expected. In the broken example, the record would show positions 1, 2, 6, 9, and 19, and the entry for position 2 is where the trouble starts.

So far find_all() has been used on a string, but the idea behind it is not tied to strings. The principle that makes this possible is that iterators and standard algorithms behave the same way on every standard container for which their use makes sense. An algorithm does not need to know which kind of container it is working with, as long as the container provides the iterators the algorithm needs.

Because of this, the source suggests that find_all() can be generalized. Instead of searching only a string for a character, the same search can be written so that it works on other standard containers. The test you read earlier would still express the same expectation in that case: every match reported should satisfy the condition being searched for. The test does not change its promise when the container changes. Only the container being searched changes.