Tokenizing Text into Words

Stopwords are words that are too frequent to be informative in text processing, so they can be discarded.

Programming

Why Frequent Words Add Little Signal

Text processing often works with large amounts of words. Many of those words do very little to tell one piece of text apart from another. Words like these show up so often that they stop being useful signals. Before a program can focus on the words that matter, it helps to have a way to set the uninformative ones aside.

Definition

Stopwords are words that are too frequent to be informative in text processing, so they can be discarded.

The question this article answers is simple: what are these words, and how can they be removed? To answer it, we will follow one example sentence through the removal process and watch what happens to each word. The sentence is the cat sat on the mat with the dog, and it is short enough to check by hand.

Tracing a Stopword Filter, Word by Word

A stopword filter works on a sentence in three steps. First, the sentence is split into words, which are the units the filter examines. Second, each word is checked against a set of stopwords. Third, only the words that are not in the set are kept. The filter checks each word on its own, so it does not look at a word's position or its neighbors.

python

sentence = "the cat sat on the mat with the dog"
stopwords = {"the", "on", "with"}
words = sentence.split()
kept = [word for word in words if word not in stopwords]
print(kept)

sentence = "the cat sat on the mat with the dog"
stopwords = {"the", "on", "with"}
words = sentence.split()
kept = [word for word in words if word not in stopwords]
print(kept)

Notes on the code

The sentence is a plain string with no punctuation, so split() returns the nine words directly.
The stopword set is kept deliberately short so each step is easy to follow.
The list comprehension keeps each word that is not in the set.

Worked example

Following each token through the filter

Apply the stopword set {the, on, with} to the nine tokens of the sentence, in order.

1
Token 1: the
The word the is in the stopword set, so it is discarded.
2
Token 2: cat
The word cat is not in the set, so it is kept.
3
Token 3: sat
The word sat is not in the set, so it is kept.
4
Token 4: on
The word on is in the set, so it is discarded.
5
Token 5: the
This is a second copy of the stopword the. It is discarded too, because the filter checks each occurrence on its own.
6
Token 6: mat
The word mat is not in the set, so it is kept.
7
Token 7: with
The word with is in the set, so it is discarded.
8
Token 8: the
The third copy of the is discarded for the same reason as the earlier copies.
9
Token 9: dog
The word dog is not in the set, so it is kept.

Result

The filter outputs cat, sat, mat, dog. Five of the nine tokens were removed, and all three copies of the word the are gone.

Which tokens survive the stopword filter, and which are removed from the sentence?

Notice that the filter removes every occurrence of a stopword, not only the first one. The word the appears three times in the sentence, and all three copies are gone from the result. The output list contains only the four words that were not in the stopword set, in their original order.

Output

['cat', 'sat', 'mat', 'dog']

Two Ways to Decide What to Discard

The source describes two main approaches for getting rid of uninformative words. Both aim at the same goal, which is removing frequent words that carry little information, but they decide which words to remove in different ways. Each approach is explained below, and the comparison table at the end of this section lists them side by side.

Applying the Frequency Rule to the Same Sentence

Now try the second approach on the same sentence. Instead of a fixed list, the program counts how often each word appears and discards any word that appears three or more times. Before reading the answer, think about which words in the sentence would be removed and which would be kept.

What do you think happens?

Using the frequency rule of discarding any word that appears three or more times, which words remain from the sentence the cat sat on the mat with the dog?

Only dog remains
cat, sat, on, mat, with, dog remain, and only the is removed
All nine tokens remain

Reveal answer

Answer:

cat, sat, on, mat, with, dog remain, and only the is removed

The word the appears three times, so it reaches the threshold and is discarded. Every other word appears only once, so each one is kept. Notice that on and with stay in the output here, even though they were removed by the fixed list in the previous section.

python

from collections import Counter

sentence = "the cat sat on the mat with the dog"
words = sentence.split()
counts = Counter(words)
kept = [word for word in words if counts[word] < 3]
print(kept)

from collections import Counter

sentence = "the cat sat on the mat with the dog"
words = sentence.split()
counts = Counter(words)
kept = [word for word in words if counts[word] < 3]
print(kept)

Notes on the code

Counter counts how many times each word appears in the list of words.
The comprehension keeps a word only when its count is below three, which removes any word appearing three or more times.
No stopword list is used, so the same sentence gives a different result from the previous example.

Output

['cat', 'sat', 'on', 'mat', 'with', 'dog']

Feature	Language-specific list	Frequency threshold
What decides removal	Membership in a fixed set of words	How often the word appears in the text
Result on the example sentence	Removes the, on, with and all their copies	Removes only the, because it appears three times
Depends on the text itself	No, the same list applies to any text	Yes, the counts change with each text
Needs a predefined list	Yes, such as a built-in English list	No, only the counts and a threshold

Two approaches to discarding uninformative words, applied to the same sentence

Mistakes When Choosing Words to Discard

Common mistake
Assuming a built-in stopword list fits every task
What it looks like
A list of English stopwords is applied to any text, including text where a word that is common in general carries real meaning for the task.
Why it fails
A list is written for general use. Whether a word is uninformative depends on the text and the task, and a list cannot know that.
Do this instead
Check the words the list removes against the actual text and decide whether each removal is safe for the goal.
Common mistake
Removing a word that changes the meaning of a sentence
What it looks like
Removing a negation such as not from a sentence changes the sentence's meaning, even though the word appears often.
Why it fails
Frequency alone does not show that a word is uninformative. A short, frequent word can still reverse the meaning of the text around it.
Do this instead
Before adding a word to a stopword list, consider whether its absence would change what the remaining words say.
Common mistake
Relying on the frequency rule to remove domain-specific terms
Why it fails
A term that appears often in a specialized text may be exactly what the task needs to find. A frequency threshold removes it because of how often it appears, not because it is unimportant.
Do this instead
Look at which frequent words the threshold removes, and keep frequent terms that matter for the subject of the text.
Common mistake
Forgetting that the filter ignores case and removes every copy
What it looks like
The sentence begins with The, but the stopword set contains only the lowercase word the. The capitalized The is kept.
Why it fails
The filter compares each word exactly as written, so a capital letter makes it a different word from the lowercase entry in the set.
Do this instead
Convert the text to a consistent case before filtering, so that every copy of a word is compared the same way.
Common mistake
Expecting the filter to use the surrounding words
Why it fails
The filter checks each word on its own. It removes every occurrence of a matching word and does not consider its position or neighbors.
Do this instead
If a removal depends on context, the filter described here cannot make that decision, and a different approach is needed.

Practice Deciding What to Discard

MEDIUM

Using the stopword set {a, and, the, to}, trace the sentence a dog and a cat ran to the park. List each token, mark it as removed or kept, and write the final output list. Then state whether the frequency rule of discarding words that appear three or more times would remove any of the same words.

Hints

Split the sentence into nine tokens first, keeping the order and every copy of a repeated word.
Two copies of a is in the sentence, and both are removed by the stopword set.
For the frequency rule, count the tokens: only one word in this sentence appears three or more times.

Your attempt (not saved)

Key Takeaways

A stopword is a word that is too frequent to be informative in text processing, so it can be discarded.
The source describes two approaches: a language-specific list of stopwords, or discarding words that appear too frequently.
A stopword filter checks each word on its own and removes every occurrence of a matching word, regardless of position.
The two approaches can give different results on the same sentence, so the choice should depend on what the removal is meant to achieve.
Common mistakes include trusting a built-in list without checking it, removing words that change meaning, and ignoring case when comparing words.

Related concepts

Working with Strings in Python

More concepts starting with T