Stop Words and Feature Selection in Text

A CountVectorizer turns every token it finds into a feature. A realistic text collection therefore produces a very large number of features, and many of them are noise: numbers, misspellings, and rare words that only show up once. This article looks at one setting, min_df, which lets you decide how rare a token may be before it is allowed to become a feature. You will make a prediction first, then run the code, then work out the reason behind the result.

The min_df parameter of CountVectorizer sets the minimum number of documents a token must appear in to become a feature. The key word is documents. min_df does not care how often a word is used overall; it asks how widely that word is spread across the collection. A token repeated many times inside a single document still has a document count of one.

The reason for this design is that a word repeated heavily in one review is a weak sign of a general pattern. The model would only have that single document to learn from. A word that shows up across many separate documents has a better chance of being a dependable signal, and min_df measures exactly that spread.

Work through the three-document corpus 'cat sat', 'cat ran', and 'dog ran' and count the documents each token appears in. 'cat' appears in documents 1 and 2, so its document count is two. 'ran' appears in documents 2 and 3, so its count is also two. 'sat' and 'dog' each appear in only one document, so their count is one. Now run the code for two settings of min_df and compare the output with your count.

The output matches the count. At min_df=1 all four tokens pass, so the matrix has four columns. At min_df=2 only 'cat' and 'ran' pass, and the matrix has two columns. The number of rows stays at three because there are still three documents. Only the columns change.

Notice the third step. Setting min_df=3 on a three-document corpus asks for a token that appears in all three documents. 'cat' and 'ran' each reach only two, so no token passes. You can derive this result by hand from the document counts alone, without running anything.

A token found in only one document is unlikely to help a model, for two connected reasons. The first is about testing. A token that appears in only one document is unlikely to appear in the test set, so it does nothing to help when the model is evaluated. The second is about learning. The model learns only from the training documents, so a feature that exists in a single training document gives it very little to generalize from.

Put simply, a token in a single document cannot show a pattern. A pattern needs the token to co-occur with other documents, or with a class label, across several examples. With only one example, there is nothing to compare it against. min_df removes these tokens, which cuts back on the feature count without giving up much that a model could use.

In the source's dataset, requiring at least five appearances of each token reduced the number of features to 27,271, which the source describes as about a third of the original features. The change was also one of kind: the source notes that there are clearly fewer numbers, and that some obscure words or misspellings have disappeared from the list.

Filtering did not make the model more accurate. The best cross-validation accuracy stayed at 89 percent, the same as before the filter. The source is direct about this: the model was not improved. The benefits it identifies are different. Fewer features can mean faster processing, and discarding useless features might make the model easier to interpret.

min_df also has a role beyond the feature count. When transform is called on a new document, it may contain words that never appeared in the training data. CountVectorizer ignores these words, since they are not part of its dictionary, and for classification this is usually acceptable because a model cannot learn anything about a word it has never seen. Some applications, such as spam detection, may still benefit from knowing how many such out-of-vocabulary words a document contains. The source suggests adding a feature that counts them, and to make that count available during training, the vectorizer has to be set up with min_df. Without it, this feature would never become active during training.