Grid Search for Model Parameters

Many text datasets contain words that show up in almost every document. Articles, pronouns, and short connectors are the usual examples. These words often carry little information about what a document is about, so a common practice is to delete them before a model sees the text. The expectation is that a cleaner feature set will help the model.

This article asks a practical question: does that expectation hold up when you measure it on a real model? Answering it well requires reading two different kinds of numbers. One number describes the data, namely how many features the model has to work with. The other describes the model, namely how well it scores. The two are easy to confuse, and the rest of this article keeps them apart.

The measurement in this article uses a logistic regression model tuned with grid search. Grid search tries a set of candidate parameter values, scores each candidate with cross-validation, and reports the best score it found. The number the script prints is therefore the best cross-validated score across the whole grid, not the score of a single fixed model.

The source snippet that produces this score assumes that X_train, y_train, and param_grid were already defined earlier in the workflow. The excerpt does not show those definitions, so the number you see is only meaningful if you know what data went into the search and which candidate values were in the grid.

In a text dataset, a feature is one column of the data the model learns from. Usually each column corresponds to a word or word-like token. Removing a stopwords list deletes the columns that match words on the list. Only the features that matched the list are affected; every other column stays as it was.

The source describes the reduction in its simplest form. Only one number changed in a meaningful way, and it changed by 305. The source describes this as most, but not all, of the stopwords having appeared in the dataset. Some words on the list never occurred in this data, so they could not remove any features.

A change in feature count and a change in model performance are separate measurements. The feature count is a fact about the data after preprocessing. The cross-validation score is a fact about how a fitted model did on held-out portions of the training data. One can change without the other, so seeing the count drop does not tell you whether the model got better or worse.

After the feature set shrank, grid search was run again and reported a best cross-validation score of 0.88. The source describes this as a slight decrease in performance. The source does not report the earlier score as a precise value, so the size of the change is best described qualitatively: the score went down a little, not dramatically.

The source gives two separate reasons the result did not justify keeping the list. First, a drop that small was judged not enough to worry about. Second, removing 305 features from more than 27,000 was unlikely to change either performance or interpretability much. When the performance change is small and the size of the feature change is small, the cost of maintaining the list can outweigh the benefit.

The source also notes when a fixed list is more useful. Fixed lists are mostly helpful for small datasets. A small dataset may not contain enough information for the model to work out which words are stopwords from the data itself. With a large dataset, the frequency patterns in the data can do much of that work.

When a result surprises you, find the exact step where the outcome departed from your expectation. This experiment has three checkpoints: the feature count before removal, the feature count after removal, and the cross-validation score after the model is refit. Compare each checkpoint with what you predicted.

This framing helps you avoid a common trap, which is to blame the whole pipeline when only one step produced the change. Checking the checkpoints in order tells you which step to investigate.

The source offers an alternative to a fixed stopwords list. Instead of naming the words to remove in advance, you can let the data decide which words are too common by using the max_df option of CountVectorizer. This option discards words that appear frequently in the dataset.

Because the threshold comes from the data, it does not depend on whether a word happens to be on someone's list. The value 0.5 in this snippet is only an illustrative setting. The source does not specify a value to use, and it does not state how the threshold is expressed. Consult the library documentation for the exact meaning of max_df before choosing a number for a real experiment.