Stop Words and Feature Filtering

CountVectorizer's min_df parameter sets the minimum frequency a word must reach to be included in the feature set.

Programming

Interactive lab

Try it: Text Vectorization: Tokens to Vectors

The text-to-tensor pipeline: standardize, split into tokens, build a frequency-ranked vocabulary with padding and [UNK], map tokens to integer ids, pad or truncate, then one-hot or multi-hot encode and look the ids up in an embedding table.

How it works

Standardize: lowercase and delete punctuation.
Split on whitespace into tokens.
Build the vocabulary: 0 = padding, 1 = [UNK], then words by frequency (ties: first appearance), capped at max tokens.
Map each token to its id (1 if it did not make the vocabulary), then pad with 0 or truncate to the sequence length.
Encode: one-hot rows per position or one multi-hot row per text; embed: id k selects row k of the embedding table.

Default run (18 steps): Raw text (59 characters): "The movie was great. The plot was great, the acting wasn't!". A model cannot read strings, so it is turned into numbers step by step. … Embedding lookup: id k selects row k of a fixed 8 x 3 table, giving a (8, 3) dense matrix instead of (8, 8) sparse one-hots. Row 0 (padding) is looked up too; a trained Embedding layer would learn these rows.

Simplified: Bounded ASCII text (up to 200 characters, 40 tokens), whitespace splitting only, and a small FIXED embedding table (a real Embedding layer learns its rows). The multi-hot vector keeps the padding slot (always 0) so its positions line up with the ids.

Educational simulation

Loading the simulation…

Why Large Vocabularies Cause Trouble

A bag-of-words model turns a collection of text documents into a matrix of word counts. Each column of that matrix corresponds to one word from the training text, and the set of those words is called the feature set. The size of the feature set matters in practice. A large vocabulary makes the matrix wide and harder to work with, and it often includes many words that appear only once or twice and contribute little to what the model can learn.

The min_df parameter of CountVectorizer gives you a way to control which words survive into the feature set. In this article you will work code-first. You will set min_df, read back the feature names, predict what should happen when you change the threshold, and only then look at the mechanism that explains the result. When a result does not match your prediction, the goal is to know exactly which line of code to examine.

Reading the Shape of a Filtered Matrix

The source fits a CountVectorizer with min_df=5 on a collection of training documents, transforms that text into a document-term matrix, and prints the matrix's shape and storage details. The variable text_train holds the training documents, loaded earlier in the source's workflow.

python

from sklearn.feature_extraction.text import CountVectorizer

vect = CountVectorizer(min_df=5).fit(text_train)
X_train = vect.transform(text_train)
print(repr(X_train))

from sklearn.feature_extraction.text import CountVectorizer

vect = CountVectorizer(min_df=5).fit(text_train)
X_train = vect.transform(text_train)
print(repr(X_train))

Output (expected)

<25000x27271 sparse matrix of type '<class 'numpy.int64'>' with ... stored elements in Compressed Sparse Row format>

Notes on the code

min_df=5 sets the threshold: a word must reach the minimum frequency to receive a column
transform builds the matrix only after the vocabulary has been decided
The printed shape reads as 25000 rows by 27271 columns

Read the output carefully. The 25000 rows correspond to the 25000 documents passed to transform. The 27271 columns correspond to the features the vectorizer kept. The phrase stored elements refers to the non-zero counts held in the sparse matrix, which is why the printed representation is a sparse matrix rather than a full table of numbers. A zero in this matrix is not stored, so the printout reports only the entries that carry a count.

Predicting the Effect of a Higher Threshold

Before reading further, decide what you expect to happen if you raise the threshold. Suppose you rerun the same code with a larger min_df value, such as 10, on the same training text. Should the number of columns go up, go down, or stay the same? Commit to an answer and a reason before you continue.

What do you think happens?

Raising min_df from 5 to 10 on the same training text: does the number of columns go up, go down, or stay the same?

Go up
Go down
Stay the same

Reveal answer

Answer:

Go down

Raising min_df removes rarer words, so fewer words qualify for a column and the number of features decreases. The source shows min_df=5 producing a 27271-column matrix, and a higher threshold yields a narrower one.

How min_df Filters the Vocabulary

Definition

min_df is the parameter of CountVectorizer that sets the minimum frequency a word must reach to be included in the feature set. A word that falls below the threshold never receives a column.

The mechanism is a decision made before the matrix is built. The vectorizer does not change how it counts words inside a document. It decides, for each candidate word, whether that word gets a column at all. Because the threshold removes rare words, the vocabulary shrinks as min_df grows. Words that are removed never appear in the transformed matrix, so their occurrences are not represented anywhere in it.

What happens to each candidate token between tokenization, frequency counting, and the final vocabulary when min_df is applied?

Tracing a Small Vocabulary by Hand

Consider a small generated corpus of three documents. Work through it by hand before running any code. The documents are: doc 1 is the movie was great; doc 2 is the movie was dull; doc 3 is a great film with a great cast.

Worked example

Counting which words reach min_df=2

Which words remain in the vocabulary when min_df=2, meaning a word must appear in at least two documents?

1
List the candidate words
The distinct words are a, cast, dull, film, great, movie, the, was, and with. That gives 9 candidates at the default min_df=1.
2
Count the documents that contain each word
the, movie, and was each appear in docs 1 and 2, so each reaches 2. great appears in docs 1, 2, and 3, so it reaches 3. a, film, with, and cast appear only in doc 3, and dull appears only in doc 2, so each stays at 1.
3
Apply the threshold
Keep only the words that reach 2: great, movie, the, and was. Drop the other five.

Result

With min_df=2 the vocabulary has 4 features, down from 9 at min_df=1.

Which words disappear from the vocabulary, and how does the total number of features change?

The before column lists the nine candidates at min_df=1. Five of them, shown as changed, appear in only one document each and are removed. The after column keeps the four words that reach the threshold. Notice that the words that survive keep their counts; the filter removes whole columns and does not alter what is counted in the remaining ones.

What do you think happens?

Using the same three documents, how many features remain when min_df=2?

Reveal answer

Answer:

4 features: great, movie, the, and was

Each of these appears in at least two documents, so each receives a column. The other five words appear in only one document and are dropped.

Inspecting Feature Names

The shape of the transformed matrix tells you how many features exist, but get_feature_names tells you which words they are. The method returns the list of feature names, one name for each column of the matrix, in column order. Storing that list in a variable lets you inspect the vocabulary, including its start, middle, and spacing, without printing all of it.

python

from sklearn.feature_extraction.text import CountVectorizer

docs = ["the movie was great",
        "the movie was dull",
        "a great film with a great cast"]

vect = CountVectorizer(min_df=2)
X = vect.fit_transform(docs)

print(X.shape)
feature_names = vect.get_feature_names()
print(feature_names)
print(feature_names[1:3])

from sklearn.feature_extraction.text import CountVectorizer

docs = ["the movie was great",
        "the movie was dull",
        "a great film with a great cast"]

vect = CountVectorizer(min_df=2)
X = vect.fit_transform(docs)

print(X.shape)
feature_names = vect.get_feature_names()
print(feature_names)
print(feature_names[1:3])

Output (expected)

(3, 4)
['great', 'movie', 'the', 'was']
['movie', 'the']

Notes on the code

X.shape gives 3 rows (documents) and 4 columns (features)
The list position of each name matches its column index in X
Slicing feature_names prints only part of the vocabulary

How does each position in the feature name list correspond to a column index in the document-term matrix?

Tracing an Unexpected Feature

When a tuned vectorizer does not behave as you expected, work through the pipeline in order rather than guessing. Each step narrows down where the divergence starts: at the parameter, at the fit, at the transform, or at the lookup. The source recommends checking the parameter value, the matrix shape, and the feature names in that order.

Check the min_df value that was actually passed to CountVectorizer. A typo there changes everything downstream.
Check the shape of the transformed matrix against the number of feature names. The column count must equal the length of the name list.
Inspect the feature names around the word you are looking for. Look at the neighbors in the list and the position where the word would sort.

Suppose a word you expected is missing from the feature names. Trace it through the same steps. First confirm the threshold value. Then count the documents that contain the word in the exact text passed to fit. If the word appears in only one document and min_df is 2, the absence is expected. If the word is present but the shape disagrees with the name list, the problem lies in how the lookup was performed.

Mistakes Beginners Make with min_df

Common mistake
Assuming a larger min_df adds features
Why it fails
Raising min_df removes rarer words, so fewer words qualify for a column. The number of features decreases.
Do this instead
Predict a smaller column count when you raise the threshold, then check the shape to confirm it.
Common mistake
Reading the shape as if it names the features
Why it fails
The shape gives only the number of columns. It does not say which words those columns represent.
Do this instead
Use the feature names list to see the actual words, in column order.
Common mistake
Expecting removed words to appear with zero counts
Why it fails
Words removed by min_df never receive a column, so they are not represented anywhere in the transformed matrix.
Do this instead
If you need a word to remain, lower min_df or check whether it is actually present in enough documents.
Common mistake
Confusing min_df filtering with a different kind of word removal
Why it fails
min_df decides based on how many documents contain a word. A stop-word list removes words by membership in a list, and the two act at different stages.
Do this instead
Decide which filter you are using, then check the threshold or the list that controls it.

Comparing Frequency Thresholds and Stop Lists

Aspect	min_df threshold	Stop-word removal
What decides removal	How often a word occurs across the training documents	Whether the word appears on a list of excluded words
Stage it acts at	Before the matrix is built, when the vocabulary is decided	Filtering of candidate tokens, based on list membership
Effect on the matrix	Removed words get no column at all	Removed words are excluded from the candidate tokens
Typical question it answers	Is this word too rare to be useful?	Is this word a common function word I want to ignore?

Two filters can shrink the feature set, but they act on different evidence.

The table makes the difference concrete. min_df looks at the data: it counts how many documents contain each word and compares that count to the threshold. A list-based filter looks at the word itself. In a vectorizer pipeline, both decisions shape the final feature set, so when the vocabulary is not what you expected, check which filter removed the word before concluding that the threshold is wrong.

Practice Tracing a Threshold Change

MEDIUM

Using the three generated documents from earlier (the movie was great; the movie was dull; a great film with a great cast), predict the feature names and the column count for min_df=3. Then run the code with that value and compare your prediction against the shape and the name list.

Hints

Count how many documents contain each word, not how many times the word occurs in total.
Only great reaches three documents in this corpus.
Compare the column count to the length of the name list to confirm your answer.

Your attempt (not saved)

Key Takeaways

The min_df parameter of CountVectorizer sets the minimum frequency a word must reach to receive a column in the feature set.
Raising min_df removes rarer words, so the number of features decreases; the source shows min_df=5 producing a 27271-column matrix.
The shape of the transformed matrix tells you how many features exist, while get_feature_names tells you which words they are, in column order.
When results diverge from expectations, check the parameter value, the matrix shape, and the feature names in that order.
Words removed by min_df never appear in the transformed matrix, so filtering happens before the matrix is built.

Related concepts

More concepts starting with S