Stopwords Removal

Text data is full of words that show up again and again but say very little on their own. Words like these are often discarded before a model is trained on the text. This article follows one experiment that does exactly that: a text dataset is processed to remove stopwords, and then the same hyperparameter search is run again. The question is simple. Did removing the stopwords change how well the model performs?

The defining feature of a stopword is frequency. These are words that turn up often across the text, so they are present in many documents. Because they appear so widely, they add bulk to the feature set without necessarily telling the model much about any particular piece of text. That is the reason they are commonly discarded before modeling. The source describes this as the usual practice, not as a rule that applies in every case, and the rest of this article explains why the decision has to be checked against results.

Before reading any score, check the size of the change. Each feature in this experiment corresponds to a term in the vocabulary, so removing stopwords changes how many features the model sees. The starting point is 27,271 features. After the stopwords are removed, 26,966 remain. Trace the change from one state to the other in the diagram below.

The source notes that the search was rerun on a dataset with 305 fewer features and that this means most, but not all, of the stopwords appeared among the features. In other words, the stopwords were present in the feature set and removing them shrank it. They did not account for every feature affected by the change, so the feature count alone does not tell the whole story about which terms were lost.

The experiment uses a grid search. A grid search tries a set of candidate hyperparameter values for a model, scores each candidate using cross-validation, and reports the best score it found. The diagram shows that loop: each candidate is tried, scored, and compared against the best result so far, and the search reports the best score at the end.

After the stopwords were removed, the search reported a best cross-validation score of 0.88. That single number is only meaningful in comparison. The source states that the grid search performance decreased slightly after removing stopwords. The honest reading is a small drop: not a dramatic one, and not an improvement. The raw value 0.88 tells you the best result of this run, while the comparison with the earlier run tells you the direction and size of the effect.

The source gives its reasoning in plain terms. The performance decrease was slight and not enough to worry about. Excluding 305 features out of more than 27,000 is unlikely to change performance or interpretability a lot. Because the effect is small in both directions, the source concludes that a fixed stopword list does not seem worth using in this case.

A fixed stopword list is a list of words written in advance and removed from the data. The source explains where such lists are most useful: mostly for small datasets. A small dataset may not contain enough information for the model to work out which words are stopwords from the data itself, so a predefined list helps. With a large dataset, the frequent words are usually visible in the data, so the model or the preprocessing step can identify them without a list written in advance.

The source also points to an alternative: discarding frequently occurring words, where the words to drop are determined from the data rather than from a list written in advance. The source text for this exercise is cut off, so the details of that approach are not described here. What the source does establish is the contrast itself: one approach reads a list from outside the data, and the other reads the frequent words from the data.