Matching Words in a String with Regular Expressions

Imagine a small program whose job is to print every whitespace-separated word in a string. It builds a regular expression, creates a regex_iterator over the string's start and end positions, and prints the first capture group of each match. The tool is neat and compact, and that neatness invites a dangerous assumption: that the iterator will report every piece of text you care about. It will not. Its results depend on exactly what the pattern allows, and the first word of a string is often silently left out.

The question to ask at every step is whether the pattern could possibly begin a match at that position. Take the text "to be or" and lay out its characters by index. The pattern R"(\s+(\w+))" needs whitespace first, then a run of word characters. The whitespace at index 2 is followed by be, so characters 2 through 4 form the first match. The whitespace at index 5 is followed by or, so characters 5 through 7 form the second match. The letters t and o at indexes 0 and 1 never take part in any match, because no whitespace comes before them.

The changed cells are the only ones the iterator reports as part of a match. Only the captured letters, be and or, are printed, because the program reads capture group 1. The whitespace cells are consumed as part of each match but never shown to the caller. Trace the program against the index map and the missing first word becomes obvious rather than mysterious.

The pattern R"(\s+(\w+))" has two parts that cooperate. The outer part, \s+, demands at least one whitespace character. The inner group, (\w+), captures one or more word characters, and that captured text is what the program prints through the index [1]. A match therefore consists of whitespace followed immediately by a word, and only the word is exposed to the caller.

Because the whitespace is part of what must match, the leading space is consumed as part of the match rather than being printed. This is why a word at the start of the string cannot be reported: there is no whitespace character available to satisfy the first half of the pattern.

The example string is illustrative and was chosen to show the mechanism; the printed result for this exact input follows from the pattern's rules as described above.

If you remove the whitespace requirement and use the simpler pattern R"((\w+))", the iterator no longer needs anything in front of a word. The source shows that this pattern does report aa, the word the first version missed. Its output for the source's test string is: aa as asd e asdf asdfg.

The simplified pattern also reports a stray letter e and the word asdf as separate matches. The input contains a caret character between them, and a caret is not a word character, so the run of word characters breaks at that point. A pattern does not know what you meant; it knows only which character classes it accepts. Each change to the pattern changes which substrings are eligible, so you must check the results against the input you actually have.

A regex_iterator reports the matches of a pattern over a range, and the source states plainly that you cannot iterate directly over an istream with one. The iterator needs a stable range with a start and an end, so the text must first reach a string or another range the iterator can accept. Only then can the iterator look for matches across the whole text.

The regex_iterator is described as bidirectional: it can move forward through matches and also move backward. Its behavior is governed by the sequence of matches the pattern produces over the range you gave it. A loop advances the iterator, reads the current match, and keeps going until the iterator equals the default-constructed value, written sregex_iterator{}. The source identifies this default value as the only possible end-of-sequence, so comparing against it is the correct way to know matching has finished.

Read the loop in the earlier code this way: each pass handles one match, the advance step moves to the next one, and the comparison against end is the single condition that stops the loop. Because the default iterator is the only end marker the source names, the loop should be written to compare against it rather than against some other guessed value.

Dereferencing a regex_iterator gives you access to the match it currently points to, and capture groups let you pick out the part you need. This is reading, and it is what the program does when it prints (*it)[1]. The iterator, however, is not a way to write values into the scanned text. The source states that you cannot write through a regex_iterator. It lets you inspect what matched; it does not give you a route for changing the text it scanned.