Feature Reduction in Text Data

Text data is full of words that appear over and over again, such as articles, prepositions, and common verbs. When a model is trained on text, every distinct word can become a feature. This article follows a single experiment built around that situation. A text dataset was processed to remove stopwords, which are common words that appear frequently in text data and are often discarded before modeling. Once those words were gone, the same hyperparameter search was run again. The question is simple: did removing the stopwords change how well the model performs?

Keep that question in mind as you read. The answer depends on two numbers that the experiment produced: how many features disappeared, and how the best cross-validation score moved. Both numbers need to be read carefully, because a large-looking change can be small in practice, and a small-looking score can still matter.

The reason for removing them is practical. If a word appears in almost every document, it adds a column to the feature set without helping the model tell documents apart. Removing those words shrinks the feature set. A smaller feature set is easier to work with, which is the motivation behind feature reduction in text data.

Consider a short sentence that has already been split into tokens: the cat sat on the mat. Suppose the stopword list used in this illustration contains the and on. Removing those two words changes the token list, and the change is easiest to see by comparing the list before and after.

Read the diagram from top to bottom. The three tokens that remain, cat, sat, and mat, carry the content words of the sentence. The three removed tokens, the, on, and the, are the common words. Notice that the word the appears twice in the original list and both copies are removed, because stopword removal works on each token, not on each unique word.

Before reading any score, it helps to know what the code in the experiment was doing. A grid search tries a set of candidate hyperparameter values for a model, scores each one using cross-validation, and reports the best score it found. The search in the experiment is the same one run before and after stopword removal, so the two results can be compared directly.

The original workflow defined several names that the search depends on, including the parameter grid and the training data. Those definitions are not shown in the source, so treat the search step itself as the object of study rather than a complete program. What matters for interpretation is the output: one best cross-validation score per run.

In this experiment, the best cross-validation score after removing stopwords was 0.88. The source describes the grid search performance as decreasing slightly after the stopwords were removed. The earlier score is not given in the source pack, so the size of the drop cannot be computed exactly here. What the source supports is the direction and the size of the change, which it calls slight.

The dataset started with 27,271 features. After the stopwords were removed, the feature count was 26,966. The first step in reading the experiment is to measure how big the change was. The pipeline below shows where the feature count is set in a typical text workflow, which explains why removing stopwords changes that number.

The order of the first three steps is an illustrative choice for this article. The source pack does not spell out this pipeline. What it does establish is that stopword removal happens before the search is rerun, and that the feature count reported afterward reflects that removal.

Two observations must be read together. The feature set lost 305 features out of more than 27,000, and the best cross-validation score dropped slightly. The source gives its reasoning in plain terms: the performance decrease was slight and not enough to worry about. Excluding 305 features out of more than 27,000 is unlikely to change performance or interpretability a lot.

The source also notes that this means most, but not all, of the stopwords appeared among the features. In other words, the stopwords were present in the feature set, and removing them shrank it, but they did not account for every feature affected by the change. This is a reminder that the feature count change and the removed stopwords are related but not identical quantities.

The experiment ends with a conclusion that goes beyond this one dataset. Because the gain from removal was small in both directions, the source concludes that it does not seem worth using a fixed stopword list in this case. The same source also explains where a fixed list is more useful, and the answer depends on dataset size.

Fixed lists are mostly helpful for small datasets. A small dataset may not contain enough information for the model to work out which words are stopwords from the data itself, so a list written in advance supplies that knowledge. With a large dataset, the frequent words are usually visible in the data, so the model or the preprocessing step can identify them without a predefined list.

The source also points to a different approach as an exercise: discarding frequently occurring words. Instead of drawing the words to drop from a list written in advance, this approach determines them from the data. The source text for this exercise is cut off, so this article does not describe its details beyond that contrast.