CountVectorizer and Bag-of-Words Features

Earlier in this chapter you saw that lemmatization maps inflected words onto a shared base form. That idea sounds tidy, but a practical question remains: does it actually make a text classifier better? Answering that question means running an experiment, not just reasoning about word forms. In this article you will commit to a prediction, trace each step of one such experiment, and then check why the outcome looks the way it does.

The experiment compares two versions of the same text representation. One is produced by the standard CountVectorizer. The other is produced after lemmatization has been applied to the text. Both versions are used to train a Logistic Regression model, and the two models are evaluated with the same cross-validation setup. The only thing meant to differ between the two runs is the vocabulary the text was reduced to. Keeping everything else fixed is what lets you attribute a difference in score to the feature change.

A vocabulary is the set of distinct words the vectorizer has seen, and each distinct word becomes one feature. Without lemmatization, every surface form is its own entry. Lemmatization changes that by mapping several inflected forms onto a single base form. After the mapping, the vectorizer counts the base form instead of each separate form, so the vocabulary shrinks.

The words in this diagram were chosen for illustration and are not taken from the experiment itself. The principle they show is the one the experiment depends on. Three surface forms of one verb would normally occupy three feature positions. After lemmatization they share one, so the feature set contains fewer distinct entries even though the underlying text has not lost any content words.

In a bag-of-words representation, each vocabulary term corresponds to one column, and each document becomes a row. The cell at a given row and column holds how many times that document contains that term. The word order is discarded, which is why the representation is called a bag of words. Because the number of columns equals the number of distinct vocabulary terms, any change to the vocabulary changes the shape of the feature matrix.

The diagram uses an invented four-term vocabulary and an invented document. Column positions here are simply labels for this illustration. What matters is the pattern: each column is one term, and the document's row records a count for each column. If lemmatization merged two of these terms into one base form, the matrix would lose one column, and every row would be counted against the remaining columns.

The experiment expects lemmatization to matter most on small datasets, and the reason is a matter of how many examples a model has to separate features that differ only slightly. When the training set is large, the model has plenty of examples to tell apart features that differ only slightly. When the training set is small, extra distinct features give the model more room to fit noise, meaning patterns that happen to appear in the few training documents but do not generalize.

This explains why the source treats merging related features as a kind of regularization. Merging removes some of the room a model has to memorize small, idiosyncratic differences between word forms. On a small dataset that restriction is useful, so the expected benefit is largest there. On a large dataset the same merge may remove distinctions the model could have used well, which is why the effect is not guaranteed to appear everywhere.

To test the small-dataset expectation, the experiment does not train on the full training data. It uses StratifiedShuffleSplit for cross-validation, with only 1 percent of the data as the training portion and the rest as the test portion. Stratification keeps the class proportions in each split similar to the full dataset. The shuffle means each repeated split draws a different random sample. The split is repeated five times, so each parameter value is judged across several small training samples rather than a single one.

The search is a grid search over the regularization strength C of Logistic Regression, with candidate values 0.001, 0.01, 0.1, 1, and 10. The grid search object is built once and then fitted twice: first on the standard CountVectorizer features, then on the lemmatized features. After each fit, the best cross-validation score is printed. Only the feature matrix at the start changes between the two fits. The splitter, the parameter grid, and the scoring are meant to stay the same.

Read the pipeline as a chain in which each stage uses the output of the one before it. A problem at any stage appears later as a strange score. The feature step is the only point where the two runs are meant to differ, so it is the first place to look when the two printed scores do not match expectations.

The two best scores in this experiment differ by 0.010 on the score scale. The source describes this as a modest improvement, and that wording deserves care. The phrase says the gap is small in absolute terms. It does not establish that lemmatization always helps. A single comparison on one dataset with one split configuration cannot settle the question, and the result is explicitly tied to the data.

When you judge the effect, ask three questions. Is the gap larger than the variation between repeated splits? Does the feature count actually drop, which shows the lemmatized matrix was built as intended? Does the result hold under a different dataset or split configuration? A modest gap that survives all three checks is a much stronger claim than a modest gap from one run.

Suppose the lemmatized score had come out lower than the baseline, or the feature count had not dropped. The useful question is where the run departed from the plan. The experiment suggests three places to check, in order.