Bag-of-Words Feature Extraction with CountVectorizer

This article starts with a prediction rather than a definition. Two text representations are built from the same training data, and each one reports its shape. The shape is a pair of numbers: the first counts rows, which are documents, and the second counts columns, which are features. Before you look at any output, try to say what each print line will report. Then check your guess against the numbers below.

A bag-of-words representation turns each document into a row of counts. Every column corresponds to one vocabulary term, and each cell records how often that term appears in that document. The word order is discarded; only the counts remain. The result is a matrix with one row per document and one column per term.

This structure explains what a column means. If two different surface forms of a word, such as a verb and its past tense, are each counted as their own term, the model must learn from each column separately. The model sees two sparse columns that carry partly overlapping meaning, and it must learn their weights independently. This is the setting in which lemmatization changes the picture.

The diagram below is an illustration built for this article, not a real output from the source. It follows three surface forms through lemmatization. Before the step, each form occupies its own column. After the step, all three forms share one base-form column, and that column pools the evidence from every form.

Notice what does not change. Each document still has a count in every remaining column, and the documents are the same documents. Only the set of columns is different. Three columns became one, so the remaining column carries the combined count of all three forms.

The reduction is a fact about the matrix shape. It does not yet say anything about accuracy. A smaller matrix is a different object from a better model, and the next section explains the reasoning behind expecting a benefit at all.

The source gives a specific reason to expect a benefit. Because lemmatization conflates some features, it works like regularization. Regularization limits how much freedom a model has to fit fine details of the training data. When distinct columns are merged, the model has fewer independent weights to adjust, so it has less room to chase patterns that are accidents of one particular sample.

That is also why the source expects the benefit to matter most on small datasets. With little training data, a model with many sparse columns has more opportunities to latch onto accidental patterns, because many columns are supported by only a few documents. Merging columns reduces that risk. On a large training set the model has enough evidence to separate the surface forms, so the merge is less likely to help much.

To test the regularization expectation, the source does not train on the full 25,000-document set. It uses only 1 percent of the data as training data and the rest as test data. The splitter is StratifiedShuffleSplit, which produces random splits, and the experiment runs five of them. The regularization strength C is tuned with GridSearchCV around LogisticRegression, so each representation is evaluated with its own tuned model.

Because the training set is so small, this setup is the case where the regularization argument predicts the largest benefit. It is a deliberate test of the expectation, not a routine evaluation of the full dataset.

The lemmatized pipeline scored 0.731 and the standard representation scored 0.721. The source calls this a modest improvement. On the score scale the gain is 0.010. It was observed in this one setup, with 1 percent training data and five random splits, so it is evidence for this experiment rather than a general guarantee.

The pack gives only the averages for the two representations. It does not report fold-by-fold scores or the spread across folds, so this source cannot tell you whether 0.010 is large compared with the variation between splits. To judge that, you would need the individual fold scores from the same run.

The source also notes that the result depends on the dataset, as many feature extraction techniques do. It says lemmatization and stemming can help build better models, or at least more compact ones. Those are two different wins. A representation may be smaller without being more accurate, and a small accuracy gain may still be worth a more compact vocabulary.

Suppose you run the first snippet and expect a smaller matrix, but the lemmatized shape prints 27271 columns. The diagram below traces the comparison pipeline in order. The checkpoint for an unexpected shape is the first stage where the lemmatized matrix exists. Check that stage before touching the model or the search.

The first print line already points to the cause. The lemmatized matrix came from lemma_vect.fit_transform, so the problem lies in that object or in the text it received. It does not lie in the standard CountVectorizer line that follows. Checking stages in order, starting with the shape of each matrix and the object that produced it, is more efficient than changing the model or the search.