Stemming Versus Lemmatization

When you turn text into numbers with CountVectorizer, every distinct word becomes a column and every document becomes a row of counts. The words are treated exactly as they appear. That means forms such as run, running, and runs are counted as three separate columns, even though they all point to the same underlying word. For a classifier or a search index, this can quietly inflate the vocabulary and split the evidence for one idea across several features.

This article answers a practical question: how do you reduce related word forms to a shared base form when you use CountVectorizer? The answer involves two separate ideas, stemming and lemmatization, and a hook that CountVectorizer provides so you can plug in your own processing. We will trace one document through that processing, read the code that builds it, and finish by learning where things go wrong when the output is unexpected.

Both stemming and lemmatization aim to reduce word forms to a shared base form. They differ in how they get there. Stemming works by chopping endings according to rules, so its output is a stem that may not be a real dictionary word. Lemmatization looks up the base form, called the lemma, which is a proper word. Because lemmatization uses word identity and context rather than suffix rules alone, it tends to produce cleaner base forms for irregular words.

Notice the difference in what you get. A stemmer can leave better unchanged because no suffix rule applies, while a lemmatizer can map it to good. Neither result is wrong in an absolute sense; each reflects a design choice. The important point for this article is that CountVectorizer performs neither operation itself, so whichever result you want must come from the text processing you hand it.

The source is explicit that scikit-learn implements neither stemming nor lemmatization as part of CountVectorizer. What it does provide is a tokenizer parameter that accepts your own callable. A callable is a function you supply. It takes a string and returns a list of tokens. CountVectorizer calls your function on each document, receives the list, and counts the items in it.

This division of labour is useful to keep in mind. CountVectorizer handles the counting, the matrix layout, and the vocabulary. Your function handles the decision about what a token is and what form it takes. When the output looks odd, you can ask which of those two responsibilities produced it.

Follow a single document as lemma_vect fits or transforms it. The path has four stages. The raw string enters the regular-expression tokenizer, which produces word-like tokens. Those tokens are then passed through spaCy's lemmatization so that each one is replaced by its lemma. The resulting list of lemmas is handed to CountVectorizer, which builds columns and counts. Read the diagram from top to bottom. The regular expression is applied before spaCy's lemmatization, not after.

Each arrow in this chain is a call made by the code. The string is the input to the first stage, and each stage's output is the input to the next. If any stage returns something unexpected, every stage after it will carry that problem forward, which is why the order of inspection matters when you debug.

The complete setup has three parts. First is a regular-expression tokenizer copied from CountVectorizer. Second is a spaCy language model whose tokenizer is swapped out for that regular-expression tokenizer. Third is a custom_tokenizer function that returns lemmas. The final line creates the vectorizer that uses that function. The source assumes that the spacy and CountVectorizer imports have already been made earlier in the course, and that text_train, a collection of training documents, is already loaded as strings.

Reading the setup, notice that it runs without transforming any document. Once the cell has run, lemma_vect exists and is ready to be fitted, but no counts have been produced yet. Keep the three parts separate in your head: the tokenizer swap, the lemma-producing function, and the vectorizer that receives the function.

The connection is a chain of hand-offs through one object. The regular-expression tokenizer is installed on en_nlp, so when custom_tokenizer passes a document to en_nlp, spaCy splits it using the copied regular expression. spaCy then runs its lemmatization on those tokens. custom_tokenizer reads the lemma_ attribute of each token and collects the results into a list. That list is the value CountVectorizer receives back from the tokenizer parameter.

A useful way to check the connection is to ask of each function, what does it receive, and what does it return? The regular expression receives characters and returns word-like tokens. The spaCy step receives those tokens and returns annotated tokens with lemmas. custom_tokenizer receives a string and returns a list of lemma strings. CountVectorizer receives that list and returns counts.

The source fits the lemmatizing vectorizer on the training text, transforms the same text, and prints the shape of the result. It then creates a standard CountVectorizer with the same min_df setting as a reference and prints its shape as well. Comparing the two printed shapes shows what lemmatization changed in the feature matrix.

Both shapes report rows and columns. The rows correspond to documents and are the same for both vectorizers, because the same training text is used. The columns correspond to the vocabulary, so any difference between the two column counts comes from the tokenizer. The source does not state the printed values, so the numbers are something you should run and observe yourself.

When a lemmatized vectorizer produces an unexpected result, debug in the same order as the data flow. Work from the earliest stage to the latest so that you find the first point where the output stops matching your expectation. Each stage depends on the ones before it, so a fault in an early stage will look like a fault in a later one.

Begin by confirming that the regular-expression tokenizer is in place on en_nlp, because every later step depends on it. Next, confirm that custom_tokenizer returns lemmas rather than raw words. Finally, confirm that both vectorizers were fitted on the same text_train. The first step where the output stops matching your expectation is the place to investigate.