Using Iterators with Regular Expressions

A reg ex_iterator reports the matches of a pattern over a range, and it cannot be pointed directly at an istream, so text should be read into a string first.

Programming

Predicting a Missing First Word

A regex_iterator reports the matches of a pattern over a range of characters, and it looks like it can be pointed at any input source and will report every piece of text you care about. That assumption causes most of the trouble people have with this tool. The iterator has real limits on what it can read, how it moves, what it lets you change, and which matches it will ever see at all.

Consider a small program whose job is to print every whitespace-separated word in a string. It builds a regular expression, creates a regex_iterator over the string's start and end positions, and prints the first capture group of each match. Read the description once and try to predict its output before reading on. The interesting question is not the printing itself but whether the first word of the text gets printed at all.

What do you think happens?

Will a whitespace-based pattern such as R"(\s+(\w+))" report a word that sits at the very start of the text?

Yes, every word is reported
No, the first word is silently skipped

Reveal answer

Answer:

No, the first word is silently skipped

The pattern requires a whitespace character before the word. A word at the start of the text has nothing in front of it to satisfy that requirement, so the iterator never begins a match there. Nothing throws an error or warns you.

Why the Iterator Cannot Read an istream Directly

A regex_iterator cannot be pointed directly at an istream. An istream is a source you read characters from in sequence, and the iterator needs a range it can scan from a start position to an end position. Because of this, the text must first be read into a string, or into another range the iterator can accept, before matching begins.

Where does the text go when it is read from a stream, and why can't the iterator walk back over characters that were already consumed?

The diagram shows the order of operations. The text must be fully available in a string before the iterator is created, because the iterator works over a range of positions. Characters that have already been pulled off a stream are not something the iterator can walk back over, so the safe route is to gather the text into a string first and then point the iterator at that string.

How a Whitespace Pattern Consumes Its Match

The pattern R"(\s+(\w+))" has two parts that work together. The outer part, \s+, demands at least one whitespace character. The inner group, (\w+), captures one or more word characters. The captured text is what the program prints through index [1]. A match therefore consists of whitespace followed immediately by a word, and only the word is exposed to the caller.

Why the Leading Word Is Never Reported

Worked example

Tracing the whitespace pattern through the text

The program uses R"(\s+(\w+))" to find words in a text whose first word begins at position zero. Which words are reported?

1
Check the first position
The text begins with a word, so there is no whitespace character before it. The pattern cannot begin a match at position zero.
2
Find the next whitespace
The iterator moves forward until it finds a position where a whitespace character is followed immediately by a word character.
3
Consume the whitespace
The whitespace is part of the match, so it is consumed and never printed. Only the word captured by (\w+) is printed through index [1].
4
Repeat for each later word
Each later word has whitespace in front of it, so each one forms a match and is reported in order.

Result

Every word that follows whitespace is reported. The first word of the text is not reported, and no error is raised.

The omission is silent. The code runs without complaint and prints a list that looks plausible, which is exactly why this bug is easy to miss. When you trace a regex_iterator by hand, ask at each step whether the pattern could possibly begin a match at that position. For this pattern, a position with no whitespace in front of it can never start a match.

Removing the Whitespace Requirement

If you remove the whitespace requirement and use the pattern R"((\w+))", the iterator no longer needs anything in front of a word. The simplified pattern reports aa, which is the word the first version missed. For the input used in the source, the output of the simplified pattern is: aa as asd e asdf asdfg.

The simplified pattern also reports a stray letter e and the word asdf as separate matches. The input contains a caret character between them, and a caret is not a word character, so the run of word characters breaks at that point. A pattern does not know what you meant; it knows only which character classes it accepts. Each change to the pattern changes which substrings are eligible, so you must check the results against the input you actually have.

Which characters does each reported match cover, and where do the matches start and end relative to the whitespace-separated tokens?

What Reading and Writing Allow

The regex_iterator is described as a bidirectional iterator, which means it can move forward through matches and also move backward. Its behavior is governed by the sequence of matches the pattern produces over the range you gave it, not by any single value you can edit. The match object it yields lets you inspect what matched, including the text and where it sits in the input.

Operation	Allowed?	What it means
Reading the matched text	Yes	Inspect the captured or matched characters through the match object
Finding where a match sits	Yes	Check the position of a match within the input range
Moving forward and backward	Yes	The iterator is bidirectional
Writing values into the scanned text	No	The iterator is not a way to change the text it scans
Reading directly from an istream	No	Input must first be placed in a string or another suitable range

What reading and writing through a regex_iterator allow and forbid

Stopping a Loop at the Default Iterator

The default regex_iterator, written sregex_iterator{}, is the only possible end-of-sequence. A loop that runs until the iterator equals this default value is the correct way to know that matching has finished. You do not count matches or check the text length; you compare the iterator against the default and stop when they are equal.

cpp

#include <iostream>
#include <regex>
#include <string>

int main() {
    std::string text = "word1 word2 word3";
    std::regex pattern(R"((\w+))");
    std::sregex_iterator it(text.begin(), text.end(), pattern);
    std::sregex_iterator end{};
    for (; it != end; ++it) {
        std::cout << (*it)[1] << '\n';
    }
    return 0;
}

#include <iostream>
#include <regex>
#include <string>

int main() {
    std::string text = "word1 word2 word3";
    std::regex pattern(R"((\w+))");
    std::sregex_iterator it(text.begin(), text.end(), pattern);
    std::sregex_iterator end{};
    for (; it != end; ++it) {
        std::cout << (*it)[1] << '\n';
    }
    return 0;
}

Notes on the code

The input is a std::string, because a regex_iterator cannot be pointed directly at an istream.
The default sregex_iterator{} is the end-of-sequence marker.
The loop stops when the iterator equals the default value.

How does a loop move from the first match to the last one, and at what point does it compare against the default-constructed iterator and stop?

Mistakes Beginners Make with Matching

Common mistake
Expecting the first word to be reported by a whitespace-based pattern
What it looks like
Using R"(\s+(\w+))" on text that begins with a word and assuming that word will appear in the output.
Why it fails
The pattern needs a whitespace character before the word, and the first word has none.
Do this instead
Use a pattern without the whitespace requirement, such as R"((\w+))", when the first word must be found.
Common mistake
Constructing the iterator directly over an istream
Why it fails
The source states that a regex_iterator cannot be pointed directly at an istream.
Do this instead
Read the input into a std::string first and create the iterator over its begin and end positions.
Common mistake
Trying to change the text through the iterator
Why it fails
The iterator lets you inspect matches, but it does not provide a way to write values into the scanned text.
Do this instead
Read matches through the iterator and build any new text in a separate string.
Common mistake
Comparing against some other value to end the loop
Why it fails
The default sregex_iterator{} is the only possible end-of-sequence.
Do this instead
Loop while the iterator is not equal to sregex_iterator{}.

Checking Your Understanding of Matches

MEDIUM

A program stores the text "alpha beta" and uses the pattern R"(\s+(\w+))" with a regex_iterator. Predict which of the two words will be printed, and explain why.

Hints

Ask whether the pattern can begin a match at the position of the word alpha.
Consider what the whitespace character before beta is doing in the match.

Your attempt (not saved)

Key Takeaways

A regex_iterator scans a range of characters, so an istream must first be read into a string before matching begins.
A whitespace-based pattern consumes the whitespace in each match, so a word at the very start of the text is never reported; a pattern without that requirement finds it.
A regex_iterator lets you read matches and move in both directions, but it does not let you write values into the scanned text.
The default sregex_iterator{} is the only end-of-sequence, so a loop should run while the iterator is not equal to it.

Related concepts

More concepts starting with U