Measuring Model Accuracy on New Data
Overfitting is a model that is too complex for its data, fitting the training set's particularities so closely that it does not generalize.
Why a Good Score on Known Data Is Not Enough
Suppose you build a model, test it on the same data it learned from, and find that it makes almost no mistakes. It is tempting to conclude that the model is excellent. The real question, though, is how it behaves on cases it has never seen. A model is only useful if its predictions carry over to new situations, and that ability is called generalization. This article explains how to judge generalization, and why the two main ways a model can fail both sit on a single dial: how complex the model is allowed to be.
What do you think happens?
A model makes almost no mistakes on the data it was built from. Does that prove it will work well on new cases?
Reveal answer
No, the real test is how it behaves on cases it has never seen.
A model is only useful if its predictions carry over to new situations. Performance on the data it learned from says little on its own about generalization.
Holding Out Data to Measure New-Data Accuracy
The only true measure of whether a method will work on new data is its performance on a held-out test set. The held-out portion is set aside and is not used to fit the model. The model learns from the training portion, and then its predictions on the held-out portion are scored. That score is the accuracy on new data, because those cases played no part in building the model.
Overfitting: Memorizing the Particulars
Overfitting is a model that is too complex for its data, fitting the training set's particularities so closely that it does not generalize.
Consider two explanations for why certain customers buy a boat. The first says that people older than 50 are the ones who buy boats. The second adds conditions about children and marital status on top of age. Both might fit the customers you already know, but the first rule is far more appealing. It uses less information, it is easier to check, and it is less likely to be tracking a coincidence in one particular group of customers. The second rule has more moving parts, and each extra part is another opportunity to capture something that is only true of the sample in front of you.
Think of a student who prepares for an exam by memorizing the answers to last year's questions word for word. On a test made of those exact questions, the student scores perfectly. Give the same student a question phrased slightly differently, or one covering the same idea from a new angle, and performance drops sharply. The student did not learn the underlying subject; they learned the particular examples. An overfitted model behaves the same way. The extra complexity gets spent on noise, accidental patterns, and individual data points rather than on the structure that would carry over to new cases.
Underfitting: Missing the Structure
Underfitting is a model that is too simple to capture the variability in the data, so it performs poorly even on training data.
Underfitting is the mirror image of overfitting. The problem is not that the model has attended too closely to the training examples. The problem is that it has not attended closely enough to the structure in them. Imagine a rule stating that everybody who owns a house buys a boat. That rule is easy to state and easy to apply, but it ignores most of what actually distinguishes buyers from non-buyers. Because it misses the variation in the data, it does badly even on the examples used to build it. Since it fails on the training set, there is no point hoping it will rescue itself on new data.
Following Complexity Along the Dial
Overfitting and underfitting are not two unrelated problems. They are two ends of one trade-off governed by model complexity. As the model is allowed to become more complex, it becomes better and better at predicting the training data, and that improvement never stops on the training side. On the new-data side, the story is different. Up to a point, added complexity helps the model pick up real structure. Beyond that point, the model starts concentrating on each individual data point in the training set, and its ability to generalize to new data falls.
Between these two regimes lies a sweet spot: a level of complexity that yields the best generalization performance. A model that is too simple misses the structure, a model that is too complex memorizes the noise, and the target lies between them. The trade-off is usually illustrated by plotting training accuracy and test accuracy against model complexity. The training curve climbs steadily, while the test curve rises, peaks, and then falls as complexity increases.
Reading the Gap Between Two Scores
Reading the three cases together makes the trade-off concrete. The too-simple case is weak everywhere. The too-complex case looks impressive on the training side, which is why it is dangerous. The sweet-spot case gives up a little training performance in exchange for the strongest result on new data, and that exchange is what generalization means in practice. The table below lists how the two scores behave in each case.
| Model complexity | Training accuracy | New-data accuracy | What the gap says |
|---|---|---|---|
| Too simple | Low | Low | Small gap, but both scores are poor |
| Sweet spot | Good | Best | Small gap, strongest result on new data |
| Too complex | High | Low | Large gap, training score hides weak generalization |
How training and new-data scores behave at each level of complexity
Interpreting a High Training Score with a Low New-Data Score
A model scores very well on its training data but noticeably worse on the held-out test set. What does this pattern indicate?
- 1Identify the two scores
The training score measures performance on the data the model learned from. The new-data score measures performance on the held-out test set.
- 2Compare them
The training score is high and the new-data score is low, so the gap between them is large.
- 3Map the gap to a failure pattern
A large gap with a strong training result is the signature of overfitting: the model fit the particulars of the training set and did not generalize.
- 4Decide what to do
Reduce complexity toward the sweet spot, then re-check the new-data score rather than trusting the training score.
The pattern points to overfitting. The model is too complex for its data, and the new-data score is the number to trust.
Common Mistakes When Judging a Model
Common mistake Treating a near-perfect training score as proof that the model is excellent
What it looks likeA model makes almost no mistakes on the data it learned from, and the builder concludes it is ready to deploy.
Why it failsThe real question is how the model behaves on cases it has never seen. Training performance alone does not measure generalization.
Do this insteadCheck accuracy on a held-out test set before drawing any conclusion.
Common mistake Assuming the most complex model is always the best choice
What it looks likeAdding more conditions to a rule because it fits the known customers better.
Why it failsEach extra part can capture a coincidence in the sample in front of you, which hurts performance on new data.
Do this insteadPrefer the simplest model that does the job, and push complexity only as far as it helps new-data performance.
Common mistake Blaming underfitting when the training score is already poor
What it looks likeTrying to fix a rule that fails on the training set by hoping it will improve on new data.
Why it failsA model that fails on its training data has not captured enough of the pattern to be useful anywhere.
Do this insteadIncrease the model's ability to capture the structure in the data before judging it on new cases.
Common mistake Confusing the training curve with the test curve when deciding where to stop
Classifying a Model from Its Two Scores
A model reaches a very high score on its training data but a much lower score on its held-out test set. Based on the gap between the two scores, which failure pattern does this most likely show, and what would you adjust first? Explain your reasoning in terms of model complexity.
Hints
- Ask which score measures generalization, and which measures performance on the data the model learned from.
- A strong training score paired with a weak new-data score is the signature described for an overly complex model.
- Think about moving the model along the complexity dial toward the sweet spot.
Key Takeaways
- Generalization, the ability to predict well on new cases, is what matters; a strong score on the training data alone does not establish it.
- Overfitting happens when a model is too complex for its data and fits the particularities of the training set so closely that it fails on new data.
- Underfitting happens when a model is too simple to capture the variability in the data, so it performs poorly even on the training data.
- Overfitting and underfitting sit at opposite ends of a single complexity dial, and the target is a sweet spot between them that gives the best new-data performance.
- Measure accuracy on a held-out test set, and read the gap between training and new-data scores to diagnose which failure pattern is present.