Regular Expression Iterators Overview

A regex iterator walks through a range of characters and reports each place where a pattern matches. Because it does this job so neatly, it is easy to assume it can be aimed at any input source and will report every piece of text you care about. That assumption is the source of most trouble with this tool. The iterator has real limits on what it can read, how it moves, what it lets you change, and which matches it will see at all. This article starts from that wrong belief and works toward the correct picture.

The iterator does not invent matches and does not look at text outside the range it was given. Its behavior is governed by the sequence of matches the pattern produces over that range. If you change the pattern, you change which substrings are eligible, and the same text can yield a completely different set of reported matches.

Read the diagram from left to right. The text is split into segments, and only the segments that start with whitespace followed by a word count as matches. The first word, first, has no whitespace before it, so no match can begin at position zero. Each later word carries its own preceding whitespace, so each one is matched. The caption describes what the iterator reports; the characters before each match are consumed as part of that match but are not what the program prints.

The pattern has two parts that work together. The outer part, \s+, demands at least one whitespace character. The inner group, (\w+), captures one or more word characters, and that captured text is what the program prints through the index [1]. A match therefore consists of whitespace followed immediately by a word, and only the word is exposed to the caller.

Because the whitespace is part of what must match, the leading space is consumed as part of the match rather than being printed. This is the reason a word at the start of the string cannot be reported: there is no whitespace character available to satisfy the first half of the pattern. Nothing in the code throws an error or warns you. The program's output is simply shorter than the input's word list, and the question to ask at each step of execution is whether the pattern could possibly begin a match at that position.

If you remove the whitespace requirement and use the pattern R"((\w+))", the iterator no longer needs anything in front of a word. The source shows that this simplified pattern reports the word the first version missed, aa, along with the other words in the input. Changing the pattern changed which substrings are eligible, and nothing else in the program had to change.

The simplified pattern also reports a stray letter e and the word asdf as separate matches. The input contains a caret character between them, and a caret is not a word character, so the run of word characters breaks at that point. The lesson is that you must check the results against the input you actually have. A looser pattern can find more matches without finding the ones you wanted.

The source states plainly that you cannot directly iterate over an istream with a regex iterator. Input must first reach a string or another range the iterator can accept. The regex iterator is built from a range of positions, so the text has to exist as a sequence of characters before matching begins. The practical flow is to read the input into a std::string first, then create the iterator over that string's begin and end positions.

The regex iterator is described as bidirectional. In practical terms, it can move forward through matches and also move backward, which separates it from iterators that only step one way. Dereferencing it gives you a match result that lets you inspect what matched and read its capture groups. Its movement, however, governs only which match you are looking at. It is not a way to write values into the scanned text.

The source identifies the default regex iterator, written sregex_iterator{}, as the only possible end-of-sequence. Each time you advance the iterator, it moves to the next match. When no further matches exist, the iterator compares equal to the default-constructed end value, and the loop should stop. Comparing against this default value is the correct way to know that matching has finished.

In the program from earlier, the loop condition it != end is exactly this comparison. The variable end is created with the default constructor, so it represents the end of the sequence. Each pass prints one captured word, then advances. The loop ends after the last match, without you counting matches or tracking positions yourself.