Tokenization in the Bag-of-Words Model
After the word is represented by its stem, both spellings point to one token.
A Vocabulary Crowded With Near-Duplicates
Imagine you have collected a large set of documents and built a vocabulary from every word that appears in them. Soon you notice something odd: the vocabulary is crowded with near-duplicates. The same underlying idea appears as a singular and a plural, or as a verb and its inflected siblings. Among the examples in the source are drawback and drawbacks, drawer and drawers, and drawing and drawings. A larger family includes replace, replaced, replacement, replaces, and replacing, which are different verb forms plus a noun connected to the verb 'to replace'.
This article is about what to do with that crowding. The question is one of standardization: when should two different surface spellings be treated as the same thing, and when does keeping them apart actually help? As you will see, the answer depends on what the model is supposed to do with those words. That dependence is the core idea behind token normalization.
Tracing a Growing Vocabulary
Follow the vocabulary as it grows. Each new surface form that appears in the documents is a new entry, so drawback and drawbacks each claim their own slot. The diagram below shows the state before any standardization: two separate entries exist, even though the two spellings carry nearly the same meaning for a bag-of-words model. After the word is represented by its stem, both spellings point to one token. Read the diagram as a mapping from each surface form (the key) to the token the model actually counts (the value).
Notice what changed. Before standardization there were two vocabulary entries, each with its own feature. After standardization there is one token, and every occurrence of either spelling feeds the same feature. The model still reads the same text; only the bookkeeping has been merged.
Why Near-Duplicates Hurt the Model
The source's argument is about the model, not about tidiness. For a bag-of-words model, drawback and drawbacks have meanings that are so close that treating them as distinct tokens adds separate features carrying nearly the same information. The source states that distinguishing them only increases overfitting and does not allow the model to fully exploit the training data.
Overfitting means the model learns patterns that belong to particular training examples rather than patterns that transfer to new text.
Connect the two ideas. When one idea is split across several vocabulary entries, the training data is spread thin across them. Each entry is seen less often, and the model has more places to memorize details instead of learning the shared meaning. The source connects this to generalization: keeping different verb forms and related words as distinct tokens works against building a model that generalizes well, in the same way that keeping singular and plural nouns apart does.
Where Normalization Sits in the Pipeline
Tokenization is the first step of the bag-of-words model. It defines what counts as a word for the purpose of feature extraction. Normalization comes after that decision. It takes the words tokenization has already produced and standardizes them, so that forms which should count together are grouped. The order matters: you cannot standardize words until you have decided what a word is.
In the diagram, tokenization decides the boundaries of the words, and normalization then reshapes those words into standardized forms. Normalization does not decide what a word is in the first place. That boundary is set earlier, by tokenization.
Two Routes to a Standard Form
Normalization is the broader goal of extracting a normal form of a word. Stemming and lemmatization are two ways of pursuing that goal, and the source describes them as the two main approaches it introduces. Both are forms of normalization. The difference between them lies in how the standardized form is found.
Stemming Versus Lemmatization Side by Side
Hold the two methods against the same word and compare what each takes as input and what it produces. The comparison below separates the mechanism from the output, which is the distinction the source draws most sharply.
| Aspect | Stemming | Lemmatization |
|---|---|---|
| Basis | Rule-based heuristic | Explicit, human-verified dictionary of known word forms |
| Uses sentence role | Not part of the described method | Yes, the word's role in the sentence is taken into account |
| Output named | Stem | Lemma |
| How the form is found | Rules that remove endings | Dictionary lookup that considers role |
Stemming and lemmatization as the source describes them
Mistakes About Normalization Scope
Common mistake Treating normalization as the step that decides what a word is
What it looks likeAssuming that standardizing forms also settles where word boundaries fall in the text.
Why it failsNormalization takes the words tokenization has already produced. Tokenization is the step that defines what constitutes a word for feature extraction.
Do this insteadDecide word boundaries first through tokenization, then apply normalization to the resulting words.
Common mistake Merging related forms regardless of the task
What it looks likeStemming every word in a corpus without asking whether its variants carry nearly the same information for the model.
Why it failsThe source's reasoning is that merging helps when the semantics of the forms are close enough that distinguishing them only adds overfitting. Whether to apply stemming, lemmatization, or neither is a judgment about the task.
Do this insteadAsk what the model is supposed to do with the words before choosing whether to merge their forms.
Common mistake Assuming a stem is always a correct dictionary form
What it looks likeBelieving that a stem produced by suffix removal must match how the word is used in a particular sentence.
Why it failsStemming is a heuristic. Its rules approximate the word and are not guaranteed to reflect its meaning or grammar in each sentence.
Do this insteadUse lemmatization when the sentence role matters and a verified dictionary form is needed.
Common mistake Confusing the stem and the lemma as interchangeable names for the same output
Predict Which Forms Merge
Consider the vocabulary entries drawer, drawers, drawing, and drawings. Using only the rule of dropping a trailing s, which pairs would merge into a single token, and which would remain separate? Explain your reasoning in terms of the model's features.
Hints
- Check each word for a trailing s and remove it where present.
- Two entries merge only when they end up with the same base.
- Think about whether the merged form loses information the model needs, using the overfitting argument from earlier.
Key Takeaways
- A bag-of-words model that keeps every surface form of a word as its own token adds near-duplicate features, which the source warns can increase overfitting.
- Tokenization decides what counts as a word; normalization then standardizes those words so that forms which should count together are grouped.
- Stemming is a rule-based heuristic that yields a stem, while lemmatization uses an explicit, human-verified dictionary, considers sentence role, and yields a lemma.
- Both stemming and lemmatization are forms of normalization, and whether to apply either depends on what the model is supposed to do with the words.