CountVectorizer and Tokenization

A CountVectorizer turns every token it finds into a feature. A token is a unit of text the vectorizer has split out of a document, such as a word. On a realistic text collection this produces a very large number of features. Many of them are noise: numbers, misspellings, and rare words that show up only once. The min_df setting lets you decide how rare a token may be before it is allowed to become a feature.

Work through the rest of this article in the same order you would work in front of a notebook: form a prediction first, run the code, and then work out why the result came out the way it did. The prediction step matters because the parameter is easy to misread if you only read its description.

The key word is documents. min_df does not count how many times a token is used overall. It counts how many separate documents contain the token at least once. A token repeated many times inside one document still has a document count of one. This is the core of the parameter: it asks how widely a word is spread across the collection, not how heavily it is used.

The reason for this design is that a heavily repeated word inside a single review is only weak evidence of a general pattern. A model that learns from that review would have only that one document to learn from. A word that shows up across many separate documents has a better chance of being a dependable signal.

Follow one small corpus through the whole process. The corpus has three documents: "cat sat", "cat ran", and "dog ran". Before any filtering happens, the vectorizer lowercases the text, splits each document into tokens, and counts the tokens inside each document. Only after that does min_df decide which tokens survive as columns.

Before running anything, predict the vocabulary at min_df=1. Ask yourself whether any token could be dropped when the threshold is a single document. The answer is no: every token in this corpus appears in at least one document, so all four tokens cat, dog, ran, and sat should remain. Now run the code to check.

Now raise the threshold to min_df=2. Each token's document count is the number of rows in which it appears. The token cat appears in the first and second documents, so its document count is two. The token ran appears in the second and third documents, so its document count is also two. The token dog appears only in the third document, and the token sat appears only in the first, so each has a document count of one and is dropped.

Compare the two matrices. Removing dog and sat deleted two columns, so the matrix shape changed from three rows by four columns to three rows by two columns. The number of rows stayed the same, because min_df never removes documents. It only removes features. Within the surviving columns, the counts are unchanged.

The model learns only from training documents. A token that appears in a single training document gives the model very little to generalize from. If the same token is absent from the test set, the model has no way to use it when it is evaluated. The source notes that a token appearing in only one document is unlikely to appear in the test set, so it is not helpful at evaluation time.

Dropping such tokens reduces the feature count without giving up much a model could use. This is what min_df does about them: it removes tokens whose document count falls below the threshold, so the matrix keeps only features that spread across more of the collection.

In one benchmark run, requiring at least five appearances of each token reduced the number of features to 27,271, which the source describes as about a third of the original features. The surviving tokens also changed in kind, not only in number. There were clearly fewer numbers in the list, and some obscure words and misspellings had disappeared. Those are the kinds of features that often appear in only one or two documents.

It is tempting to assume that removing noisy features will improve a model. The source is direct about this. In its benchmark, the best cross-validation accuracy stayed at 89 percent, the same as before the filter. The model was not improved by min_df. The benefit lies somewhere else.

The benefits the source identifies are faster processing, because there are fewer features to handle, and a model that may be easier to interpret, because useless features have been discarded. Filtering is therefore a decision about cost and clarity, and you should check accuracy yourself rather than assume it changed.

Some applications, such as spam detection, may benefit from knowing how many out-of-vocabulary words a document contains. The source suggests adding a feature that counts them. To make that count available during training, the vectorizer has to be set up with min_df. Without it, this feature would never become active during training.