Filtering Rare Words with min_df

When a computer turns a document into numbers, every distinct word it sees can become a feature. Words such as 'the', 'and', 'is', and 'of' appear in almost every English document, so they tell a model very little about what any single document is about. They still occupy a column of the feature matrix, though, and they make the representation larger.

This section looks at a practical response to that problem: a pre-defined stopwords list. You will see what such a list contains, how it is passed to CountVectorizer, and what changes in the resulting features. Before reading each explanation, try to trace the program in your head and decide what you expect each stage to produce. Then compare your expectation with what the code actually does.

It helps to see the stopword step as one stage in a sequence rather than as a standalone operation. Text is first split into tokens. The stopword filter then removes the tokens that appear in the list. The remaining tokens form the vocabulary, and the count matrix records how often each vocabulary item appears in each document. This is a simplified conceptual model. It is meant to show where the stopword decision sits, not to describe every internal detail.

Follow the example sentence through the diagram. It starts as one raw string. It becomes four tokens. The stopword filter then decides, token by token, which ones survive. Only the survivors can become vocabulary entries, and only the vocabulary entries get columns in the count matrix. A surprising feature count therefore has to come from one of these four stages, and the stopword filter is the stage where the list you supply makes its decisions.

The scikit-learn library provides this list in its feature extraction text module under the name ENGLISH_STOP_WORDS. The sampled entries look ordinary. They include pronouns such as 'he' and 'me', determiners such as 'the', and connecting words such as 'although' and 'into'. The sample contains no subject-specific vocabulary. The list is built around words that carry grammatical function rather than meaning about a particular topic, which is why it is a generic starting point for English text.

You can import the list directly and ask how many entries it holds. Because the list is a set-like collection rather than an ordered sequence, convert it to a list before slicing it. The slice below takes every tenth entry, which is a quick way to sample the contents without printing all 318 words.

CountVectorizer transforms text into a matrix of token counts. It accepts a stop_words parameter. When you set that parameter to the string 'english', the vectorizer uses the built-in list. The setting is supplied when the object is created, and the vocabulary is learned when fit is called on the training text. The transform step that follows fit uses the vocabulary and stopword handling that were set up earlier.

A small illustration makes the effect concrete. Suppose a document contains the words 'the', 'movie', 'was', and 'fun', and the list contains 'the' and 'was'. The vectorizer would not treat 'the' and 'was' as features, so the document is represented only by the words that remain. The illustration uses invented words to show the principle. The real effect depends on the words in the list you choose.

Notice what changes and what does not. The dropped words are the ones that were removed. The remaining words keep their order and become the candidates for the vocabulary. The removal is applied to the words of the text, not to the documents themselves. Each document still exists, just with fewer candidate features.

The feature count can only shrink or stay the same when stopwords are removed. The source adds a more hopeful claim: although the count can only fall, the performance of a downstream model might improve. Whether it does is an empirical question. The source frames it as something to test by trying the change and measuring the result.

Each surviving word in the vocabulary occupies one column of the count matrix. When a stopword is removed, the words after it move to new column positions, so a column number that meant one word before filtering may mean a different word afterward. This is why you should read feature names from the fitted vectorizer rather than assuming a fixed column order.

In the diagram, the two survivors take columns 0 and 1 once the list is applied, even though movie and fun sat in later positions among the original four candidates. The exact indices depend on the vocabulary the fitted vectorizer produces, so verify them with the feature names rather than trusting a hand count.