Text Features with CountVectorizer

A common habit in text modeling is to delete words that carry little meaning before a model sees the text. Articles, pronouns, and short connectors such as these appear in almost every document, and they often say little about what a document is actually about. The expectation is that a cleaner set of features will help the model. This article asks a practical question: does that expectation hold up when you measure it on a real model?

A feature, in this setting, is one column of the data the model learns from. In a text dataset, each column usually corresponds to a word or word-like token. The set of columns is the vocabulary the model can see. When you tell the pipeline to remove a stopwords list, it deletes only the columns whose words match that list. Every other column stays exactly as it was.

Think of the feature set as a table with one column per word. A dataset with 27,271 features has 27,271 such columns. Removing stopwords does not shorten the documents or change how often the remaining words occur. It only removes the columns that correspond to words on the list. Columns for words not on the list are untouched.

Before looking at numbers, trace the path a document takes. Raw documents are split into tokens. Each token is then checked against the filter you chose, either a stopwords list or a frequency rule such as max_df. The tokens that pass the filter become the columns of the document-term matrix, and the model learns from that matrix. Each stage is a checkpoint where the outcome can be compared with your expectation.

The source notes that the stopwords list covered most, but not all, of its words in this dataset. Some words on the list did not occur in the data at all, so they could not remove any features. The 305 removed features are therefore the words from the list that did appear, and the reduction reflects only those.

The experiment uses a logistic regression model tuned with grid search. Grid search tries a set of candidate parameter values, scores each one with cross-validation, and reports the best score it found. The script reports one number, the best cross-validation score, and that number is only part of the story. You should first check what changed in the data itself, because the score is only meaningful once you know how the feature set moved.

The feature count is a property of the data. The cross-validated score is a property of the trained model's measured performance. The stopwords step affects the first directly and affects the second only indirectly, through the model that is refit on the smaller feature set. Because they are two different measurements, a small change in one does not automatically explain the other. This is why the three checkpoints must be checked in order.

After the feature set shrank, grid search was run again and reported a best cross-validation score of 0.88. The source describes this as a slight decrease in performance. The source does not report the earlier score as a precise value, so the size of the change is best described qualitatively: the score went down a little, not dramatically.

The source gives two separate reasons the result did not justify keeping the list. First, a drop that small was judged not enough to worry about. Second, removing 305 features from more than 27,000 was unlikely to change either performance or interpretability much. When a change is small in performance and also small in the size of the feature set, the cost of maintaining the list can outweigh the benefit.

The source notes that fixed lists are mostly helpful for small datasets. A small dataset may not contain enough information for the model to work out which words are stopwords from the data itself. With a large dataset, the frequency patterns in the data can do much of that work. In this experiment the dataset was large, which is part of why the fixed list added little.

Instead of naming the words to remove in advance, you can let the data decide which words are too common. The max_df option of CountVectorizer discards words that appear frequently in the dataset. Because the threshold comes from the data, it does not depend on whether a word happens to be on someone's list.