Stemming and Lemmatization Compared

This article starts with a prediction rather than a definition. Two text representations are built from the same training data, and each one reports its shape. A shape has two numbers: the first counts rows, which are documents, and the second counts columns, which are features. Before you read any output, decide what you expect each number to be for the lemmatized representation compared with the standard one.

A shape line is the fastest check on any vectorizer. The first number tells you how many documents went in, and the second tells you how many vocabulary columns the vectorizer learned. Because the vectorizer creates one column per vocabulary term, the second number is a direct measure of how many distinct terms the representation keeps track of. When you change the text processing, the shape is the first thing to inspect, and it tells you whether the change did what you intended.

In a bag-of-words matrix, each column counts how often one vocabulary term appears in each document. If two different surface forms of the same word are both counted, they occupy two separate columns, and the model must learn from each column on its own. Lemmatization reduces words to their base or root form. When several surface forms map to one base form, their counts land in one shared column. The matrix ends up with fewer columns, and each remaining column pools evidence from several surface forms.

The three forms in the diagram are an illustration chosen for this article, not a figure taken from the source's dataset. The mechanism is the same for any group of forms that share a base form: each form that used to own a column now contributes its counts to the single base column.

The source gives exact numbers for the effect. The standard representation has 27,271 features, and the lemmatized representation has 21,596. The difference is 5,675 features. Expressed as a share of the original columns, 5,675 out of 27,271 is roughly 21 percent. The number of documents stayed at 25,000 in both cases, so the entire change is in the columns.

A smaller feature set is not automatically a better model. The source gives a specific reason to expect a benefit: because lemmatization conflates some features, it works like regularization. Regularization limits how much freedom a model has to fit fine details of the training data. When two forms that carry nearly the same meaning are merged, the model has fewer separate weights to estimate, and it cannot assign a different importance to each surface form unless the data supports that.

This is why the source expects the benefit to be largest when the dataset is small. With little training data, a model with many sparse columns has more opportunities to latch onto accidental patterns, such as a rare form that happens to appear only in one class. Each column has only a few documents supporting it. Merging columns pools those few documents across several forms, so each remaining column rests on more evidence. That reduces the risk of fitting noise.

To test the small-dataset expectation, the source does not use the full 25,000-document training set. It builds a cross-validation experiment that uses only 1 percent of the data as training data and the rest as test data. The splitter is StratifiedShuffleSplit, and the search over the regularization strength C is run with GridSearchCV around LogisticRegression. The comparison uses five random splits.

The gain is a difference of 0.010 on the score scale. Relative to the standard score of 0.721, that is about 1.4 percent. The source calls this a modest improvement. It was observed in this specific setup, with 1 percent training data and five random splits, so it is evidence for that experiment rather than a general promise. The source pack reports the two averages but does not give per-fold scores, so the spread across individual splits cannot be read from it.

The source also notes that lemmatization and stemming can sometimes produce a better model, or at least a more compact one. These are two different wins. A representation can be smaller without being more accurate, and a small accuracy gain can still justify a compact vocabulary. When you report a result, state which of the two you observed.

Suppose you run the first snippet and expect a smaller matrix, but the lemmatized shape prints 27,271 columns. Do not change the model or the search yet. Work through the pipeline in order and compare each stage's output against what you expected. The chain runs from raw text, through lemmatization and vectorizing, to fitting and cross-validating. Each stage hands its output to the next, so an error at one stage will carry forward into everything after it.

The first print line already tells you where to look. It was produced by lemma_vect.fit_transform, so the problem lies in that object or in the text it was given, not in the standard CountVectorizer line that follows. Check whether the text passed in was actually lemmatized, and whether the vectorizer was built on that processed text. Only after the shapes agree with your expectations should you move on to the search and the scores.