Measuring Model Performance on Text Features

When a computer turns a document into numbers, every distinct word it sees can become a feature. Words such as 'the', 'and', 'is', and 'of' appear in almost every English document. Because they show up everywhere, they tell a model very little about what any single document is about. They still occupy a column of the feature matrix, though, and they make the representation larger than it needs to be.

This article looks at a practical response to that problem: a pre-defined stopwords list. You will see what such a list contains, how it is passed to CountVectorizer, and what changes in the resulting features. Before reading each explanation, try to trace the code in your head and decide what you expect each stage to produce. Then compare your expectation with what the code actually does.

The scikit-learn library provides this list in its feature extraction text module, under the name ENGLISH_STOP_WORDS. You can import it directly and ask how many entries it holds. The source reports that this list contains 318 stopwords. The list is set-like rather than an ordered sequence, so the source converts it to a list before slicing it. The slice in the source takes every tenth entry, a quick way to sample the contents without printing all 318 words.

The sampled words look ordinary. The source mentions pronouns such as 'he' and 'me', the determiner 'the', and connecting words such as 'although' and 'into'. The sample contains no subject-specific vocabulary. The list is built around words that carry grammatical function rather than meaning about a particular topic, which is why it is a generic starting point for English text. Because it is generic, it may not suit every corpus, and that is one reason the effect of using it has to be checked.

It helps to see the stopword step as one stage in a sequence rather than as a standalone operation. Text is first split into tokens. The stopword filter then removes the tokens that appear in the list. The remaining tokens form the vocabulary, and the count matrix records how often each vocabulary item appears in each document. This is a simplified conceptual model. Its purpose is to show where the stopword decision sits, not to describe every internal detail.

Notice that the removal is applied to the words of the text, not to the documents themselves. Each document still exists, just with fewer candidate features. The changed words are the ones that were dropped, and the remaining words keep their order as they become candidates for the vocabulary.

CountVectorizer transforms text into a matrix of token counts. It accepts a stop_words parameter. When you set that parameter to the string 'english', the vectorizer uses the built-in list. The parameter is given when the object is created, and the vocabulary is then learned when fit is called on the training text. The transform step that follows fit uses the vocabulary and stopword handling that were set up earlier.

The source gives a small illustration. A document contains the words 'the', 'movie', 'was', and 'fun', and the list contains 'the' and 'was'. The vectorizer would not treat 'the' and 'was' as features, so the document is represented only by the words that remain. The example below applies the same idea to two short sentences.

The logic behind the worked example is simple. Removing stopwords is a filter applied to candidate words. A filter can keep or discard a word, but it cannot introduce one that was not there. So the feature count can only shrink or stay the same. The most it can drop is by the length of the list, which is 318 for ENGLISH_STOP_WORDS.