Overfitting and Generalization in Text Models
The answer depends on what the model is supposed to do with those words.
A Vocabulary Crowded With Near-Duplicates
Suppose you have collected a large set of documents and built a vocabulary from every word that appears in them. Very quickly you notice a problem: the vocabulary is crowded with near-duplicates. The same underlying idea shows up as a singular and a plural, or as a verb and its inflected siblings. A word such as drawback appears alongside drawbacks, drawer alongside drawers, and drawing alongside drawings. A larger family appears around the verb to replace, which gives replace, replaced, replacement, replaces, and replacing.
The question this article takes up is what to do with that crowding. It is a question about standardization: when should two different surface spellings be treated as the same thing, and when does keeping them apart actually help? The answer depends on what the model is supposed to do with those words. That dependence is the core idea behind token normalization, and it is the thread we will follow from here.
Tracing Vocabulary Entries Before Any Merging
What do you think happens?
If a bag-of-words model treats drawback and drawbacks as two distinct tokens, how many separate features does it end up with for that idea?
Reveal answer
Two features, each counted separately even though their meanings are nearly the same
Each distinct surface form becomes its own vocabulary entry. The source describes the result as separate features that carry nearly the same information.
Read the diagram from left to right. On the left, drawback and drawbacks each hold a vocabulary entry of their own, so the training data for that single idea is divided between them. On the right, both spellings point to one shared token. The idea itself has not changed, but the vocabulary now has one place to record it instead of two.
Counting Vocabulary Entries for the Drawer Family
A collection of documents contains drawer, drawers, drawing, and drawings. How many vocabulary entries does the model hold if every surface form is kept distinct, and how many distinct underlying ideas are there?
- 1List the surface forms
The collection contains four distinct spellings: drawer, drawers, drawing, and drawings.
- 2Count the entries without merging
Each distinct spelling gets its own vocabulary entry, so there are four entries.
- 3Group by shared idea
The source pairs drawer with drawers as singular and plural, and drawing with drawings in the same way. That gives two underlying families, one for each pair.
- 4Compare the two counts
Four entries cover only two underlying ideas, so each idea is spread across two entries and its training examples are divided between them.
Four vocabulary entries represent two underlying ideas. Keeping the forms apart splits the training evidence for each idea across its variants.
Why Split Entries Lead to Overfitting
Overfitting means that the model learns patterns that belong to the particular training examples rather than patterns that transfer to new text. A model that overfits has latched onto details of the examples it was shown, and those details do not carry over to documents it has not seen.
Splitting one idea across several vocabulary entries makes this more likely. The training data is spread thin across those entries. Each entry is seen less often than the combined idea would be, and the model has more separate places to memorize specific details instead of learning the meaning the variants share. Each extra entry is a new feature that carries little new information, because its meaning is already covered by its siblings.
The source connects this to generalization. Generalization is the ability to perform well on new text, and the source argues that treating different verb forms and related words as distinct tokens works against it. It makes the same point about singular and plural nouns: keeping them apart is disadvantageous for building a model that generalizes well.
The source states that for a bag-of-words model, distinguishing drawback from drawbacks will only increase overfitting, and will not allow the model to fully exploit the training data. Merging them is therefore a way of pooling evidence: the model sees one richer entry where it would otherwise see two thinner ones.
Conflating Forms Into One Stem
The source's remedy is to represent each word by its word stem. Representing a word this way identifies, or conflates, all of the words that share the same stem, so that related forms become one standardized token instead of several separate ones. Once conflated, drawback and drawbacks are no longer two separate vocabulary entries. They are two surface forms that the model reads as one unit.
The diagram shows the replace family as the source describes it: several different verb forms plus a noun connected to the verb to replace. Each of these surface forms arrives at the same stem token. The vocabulary no longer needs a separate entry for each spelling, and the training occurrences of every member of the family add to one shared count.
When you decide whether to conflate forms, ask whether the variants carry nearly the same meaning for the task. If they do, merging them pools evidence. If the distinction matters for what the model must do, keeping the forms apart may be the right choice.
Stemming Versus Lemmatization
Normalization is the broader goal of extracting some normal form of a word. Stemming and lemmatization are the two main approaches the source introduces for pursuing that goal. Both are forms of normalization, and the difference between them lies in how the standardized form is found and what it is called.
| Aspect | Stemming | Lemmatization |
|---|---|---|
| Basis of the method | Rule-based heuristic, such as dropping common suffixes | Explicit, human-verified dictionary of known word forms |
| Information used | The shape of the word | The dictionary and the word's role in the sentence |
| Resource needed | Rules only, not a store of known word forms | A dictionary of known word forms |
| Output named by the source | A stem | A lemma |
| Status of the result | A practical approximation that is not guaranteed to reflect meaning or grammar in each sentence | Entries checked by people rather than generated by a rule |
Stemming and lemmatization as the source describes them
Where Normalization Fits in the Pipeline
The source describes tokenization as the first step of the bag-of-words model. Tokenization defines what constitutes a word for the purpose of feature extraction. Normalization comes after that decision. It takes the words that tokenization has already produced and standardizes them, so that forms which should count together are grouped. The order is therefore raw text, then tokenization, then normalization, and only then the vocabulary the model uses.
Whether to apply stemming, lemmatization, or neither is a judgment about the task. The source's reasoning is that merging related forms helps when their semantics are close enough that distinguishing them only adds overfitting. When the meanings differ in ways the task depends on, merging would throw away information the model needs.
Mistakes Beginners Make With Normalization
Common mistake Assuming that more distinct vocabulary entries always means a richer model
What it looks likeKeeping drawback and drawbacks as separate tokens and expecting the model to gain information from the split.
Why it failsThe source argues that the two forms have such close meanings that the separate entries add features carrying nearly the same information, which increases overfitting.
Do this insteadAsk whether the distinct forms carry different meaning for the task before keeping them apart.
Common mistake Treating stemming and lemmatization as interchangeable
What it looks likeAssuming that a stem and a lemma are the same kind of output because both come from normalization.
Why it failsStemming uses a rule-based heuristic that looks at word shape and yields a stem, while lemmatization uses a human-verified dictionary, considers sentence role, and yields a lemma.
Do this insteadName the method and its output separately, and choose according to whether the sentence role of the word matters for the task.
Common mistake Believing a stemming rule is guaranteed to be correct
Why it failsThe source calls stemming a heuristic, which is a practical approximation. Its rules are not guaranteed to reflect the meaning or grammar of the word in each sentence.
Common mistake Placing normalization before tokenization
What it looks likeTrying to standardize forms before deciding where the words begin and end.
Why it failsThe source describes tokenization as the first step, defining what counts as a word, and normalization as coming after it.
Do this insteadRun tokenization first, then normalization, then build the vocabulary.
Practice Deciding What to Merge
A bag-of-words model is built from a set of documents that contain the words replace, replaced, replacement, replaces, and replacing. Describe how representing each word by its word stem changes the vocabulary, and state one reason the source gives for why merging these forms can help the model. Then explain whether the merge would still be appropriate if the task depended on whether a particular occurrence of the word was used as a verb or a noun.
Hints
- Go back to the key-value map: several surface forms map to one shared stem token.
- The source's reason is that the meanings are close enough that distinguishing them only adds overfitting.
- Consider what lemmatization adds that stemming does not, and whether the sentence role matters for this task.
Key Takeaways
- Several surface forms of one word, such as drawback and drawbacks, add vocabulary entries without adding much new meaning, which spreads the training data thin and invites overfitting in a bag-of-words model.
- Representing each word by its word stem conflates all words that share that stem into one standardized token, so the model reads related forms as one unit.
- Stemming is a rule-based heuristic that yields a stem, while lemmatization uses an explicit, human-verified dictionary, considers the word's role in the sentence, and yields a lemma.
- Tokenization comes first and defines what counts as a word; normalization comes after and standardizes those words, and it does not decide word boundaries.