Implementing Lemmatization in Text Preprocessing

This article starts with code instead of theory. The excerpt below builds two text representations of the same training documents and prints the shape of each one. Shape tells you how many rows and how many columns a matrix has. In this setting, rows are documents and columns are features. Before you read the output, try to say what each print line will report. Your prediction is the point of the exercise, because the difference between what you expected and what the output shows is where the learning happens.

A bag-of-words matrix has one column for each vocabulary term. Each column counts how often that term appears in each document. So the number of columns is the number of features the model can use. When you change how words are normalized before counting, you change the vocabulary, and the column count follows. Shape output is therefore a fast check on what preprocessing actually did.

Look at what changed and what did not. The number of columns dropped, and the number of rows, which counts documents, did not change at all. Lemmatization does not remove documents or alter any document's text length in the matrix sense. It changes the set of column labels, so that several surface forms of a word share one column.

Lemmatization reduces words to their base or root form. Different surface forms of the same word, such as running, runs, and ran, can all map to one base form. Once that mapping is applied before counting, the three forms no longer each get their own column. Their counts pool into a single column for the base form. The mapping is many-to-one: several keys point to one value.

Read the diagram as a vocabulary reduction. Before lemmatization, the three forms would be three separate columns, each counting its own occurrences. After lemmatization, the counts for all three accumulate in one column. Multiply this pattern across a whole corpus, and it accounts for the drop from 27,271 to 21,596 features in the source's example. Each remaining column now pools evidence from several surface forms.

A smaller feature set is not automatically a better model. The source gives a specific reason to expect a benefit: because lemmatization conflates some features, it works like regularization. Regularization limits how much freedom a model has to fit fine details of the training data. Merging columns that carry overlapping meaning means the model has fewer separate knobs to tune to accidental patterns.

The effect is expected to matter most when the dataset is small. With little training data, a model that has many sparse columns has more chances to latch onto patterns that appear only by chance. A word that shows up in two or three training documents gives the model very little evidence, yet it still gets its own column. Merging those rare forms into a shared column pools their evidence and reduces that risk. With a large training set, there is more evidence per column to begin with, so the regularizing effect of merging is expected to matter less.

To test the small-dataset expectation, the source does not train on the full 25,000-document set. It runs a cross-validation experiment that uses only 1 percent of the data as training data, with the rest serving as test data. The splitter is StratifiedShuffleSplit, which produces five random splits. Inside each split, GridSearchCV searches over the regularization strength C for a LogisticRegression model. Both the standard CountVectorizer arm and the lemmatized arm go through this same search, so the comparison isolates the effect of the preprocessing step.

Notice that the two arms differ in exactly one stage: how the text is tokenized and normalized before counting. The splitter, the search, and the classifier are the same. That is what lets you attribute the score difference to the preprocessing rather than to the model setup. The table also shows the two wins the source mentions: the lemmatized arm uses fewer columns and scores slightly higher.

The lemmatized pipeline scored 0.731 against 0.721 for the standard representation. The gain is a difference of 0.010 on the score scale. The source calls this a modest improvement. Read the number carefully: it is a real difference in this experiment, but it is small, and it was observed under one specific setup of 1 percent training data with five random splits.

The source is also careful about generality. It states that the result varies depending on the dataset, as many feature extraction techniques do. It notes that lemmatization and stemming can sometimes help build better, or at least more compact, models. So there are two possible wins to check separately: accuracy and compactness. A small accuracy gain may still be worth a much more compact vocabulary, and a compact vocabulary may give no accuracy gain at all.

Now return to the opening code. Suppose you expected a smaller matrix from the lemmatized arm, but its shape prints 27,271 columns, which matches the standard arm. Where should you look? Work through the pipeline in order and check each stage's output against what you expected, starting with the earliest stage that could have gone wrong. Do not change the model or the search first. The first print line already points to the cause: the lemmatized matrix was produced by lemma_vect.fit_transform, so the problem lies in that object or in the text it was given, not in the standard CountVectorizer line that follows.

The diagram shows why the order matters. The shape check after lemma_vect.fit_transform is the first place the discrepancy can be seen. Everything downstream, including the split, the grid search, and the scores, receives whatever that stage produced. If the lemmatized matrix is wrong at its first output, every later score is built on the wrong features, so comparing scores before checking shapes would waste effort.