Text Tokenization with Regular Expressions

Notice that the regular expression is applied before spaCy's lemmatization, not after.

Programming

Interactive lab

Try it: Text Vectorization: Tokens to Vectors

The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.

How it works

Standardize: lowercase and delete punctuation.
Split on whitespace into tokens.
Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.

Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.

Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.

Educational simulation

Loading the simulation…

Why Word Forms Split the Vocabulary

When you turn text into numbers with CountVectorizer, every distinct word becomes a column and every document becomes a row of counts. The words are treated exactly as they appear. That means forms such as run, running, and runs are counted as three separate columns, even though they share a common base form.

Lemmatization is one way to reduce these related forms to a shared base form. This article shows how to add lemmatization to CountVectorizer, and it does so by working through the code path one document takes rather than by memorizing a single function.

EASY

Before reading further, think about the question directly. If two words in a document share a base form, how many columns could they occupy in a standard CountVectorizer? Write down your answer. Then ask yourself whether a lemmatized vectorizer would end up with more columns or fewer columns than a standard one. The rest of the article explains the mechanism that determines the answer.

Hints

Think about what a column represents: one distinct string value in the vocabulary.
Think about what happens to inflected forms once they are reduced to a shared base form.

Your attempt (not saved)

The Tokenizer Parameter as a Hook

CountVectorizer does not perform stemming or lemmatization as part of its own behavior. This is a deliberate boundary of the library rather than a bug to work around: the vectorizer counts whatever tokens it is given. What it does provide is a tokenizer parameter that accepts your own callable.

Definition

A tokenizer callable, in this context, is a function that takes one string and returns a list of tokens. CountVectorizer calls it on each document and counts whatever list comes back.

Key point

The tokenizer parameter is the hook. If you supply a function that returns lemmas instead of raw words, CountVectorizer will count those lemmas. The vectorizer itself does not need to know that lemmatization happened.

In the real world

Consider three forms of one verb: running, ran, and runs. A standard vectorizer gives each of them its own column. A tokenizer callable that returns lemmas for these three words would hand the vectorizer the same base form each time, so all three occurrences would be counted in one shared column. This is the mechanism the rest of the article builds, and the forms used here are an illustrative choice rather than an example taken from the source.

Tracing One Document Through the Pipeline

The diagram below shows the path one document takes when the lemmatizing vectorizer fits or transforms it. Read it from top to bottom. Notice that the regular expression is applied before spaCy's lemmatization, not after. The order is what makes the rest of the construction work.

What happens to one raw document, step by step, as it becomes a list of lemmas that CountVectorizer counts

Reading the Setup in Order

The full setup has three parts: a regular-expression tokenizer copied from CountVectorizer, a spaCy language model whose tokenizer is swapped out, and a custom_tokenizer function that returns lemmas. A final line creates the vectorizer that uses that function. Read the parts in that order, because each one depends on the one before it.

Setup of this kind does not transform any documents by itself. Once the setup has run, the lemmatizing vectorizer is ready to be fitted, but nothing has been fitted yet. The source does not show output for this step, because no transformation has happened at this point.

Where CountVectorizer Hands Off Its Work

The construction has two halves that are easy to confuse. The first half changes how spaCy splits a string into tokens. The second half uses spaCy to produce lemmas from those tokens. The two halves are connected through the same spaCy object, en_nlp. The sequence below shows which component calls which when a document is processed.

Where CountVectorizer hands off to the custom tokenizer, and what it does with the list that comes back

Comparing the Two Vectorizers' Shapes

The source fits the lemmatizing vectorizer on the training text, transforms the same text, and prints the shape of the result. It then creates a standard CountVectorizer with the same min_df setting as a reference and prints that shape too. Comparing the two printed shapes is how you see what lemmatization changed in the feature matrix.

Both shapes report a number of rows and a number of columns. The rows correspond to documents and are the same in both cases, because the same training text is used. The columns correspond to the vocabulary. Any difference between the two printed column counts therefore comes from the tokenizer, since that is the only thing that differs between the two vectorizers in this comparison.

How inflected forms each map to one base form, and how that reduces the number of distinct vocabulary entries

Debugging From the Start of the Data Flow

When a result looks wrong, debug in the same order as the data flow. Every later step depends on the earlier ones, so checking the last step first wastes effort. The flowchart below lists the checks in order, and the first check that fails marks where the result stops matching what you expected.

The checks to run in data-flow order, ending at the first step whose output does not match expectations

Notice that the first check is whether the regular-expression tokenizer is in place on en_nlp. If that swap did not happen, the lemmas that custom_tokenizer returns come from spaCy's default word boundaries, so the vocabulary can differ from the one you expect even though every other line looks correct.

Applying the Chain to a New Sentence

MEDIUM

Suppose a custom_tokenizer function is correct, but a lemmatized vectorizer produces the same column count as a standard vectorizer on the same text. Using the data-flow order from this article, decide which check you would run first and explain why the other checks come after it.