Model Complexity and Dataset Size
Overfitting is a model that is too complex for its data, fitting the training set's particularities so closely that it does not generalize.
Why a Perfect Score Can Mislead
Suppose you build a model, test it on the same data it learned from, and find that it makes almost no mistakes. It is tempting to conclude that the model is excellent. The real question, though, is how it behaves on cases it has never seen. A model is only useful if its predictions carry over to new situations. The ability to do that is called generalization, and it is the thread that runs through this whole article.
Two common failure patterns explain most of the ways a model can look fine in one setting and disappoint in another. They are called overfitting and underfitting. They sit at opposite ends of a single dial: how complex the model is allowed to be. Understanding that dial is the key to building models that work on data nobody has shown them yet.
Generalization: The Question That Matters
A training score tells you how well a model handles the examples it was built from. It does not, by itself, tell you how the model will handle a new customer, a new image, or a new record. The only true measure of whether a method will work on new data is its performance on a held-out test set, meaning examples that were kept back and not used to build the model.
Practitioners also lean on simplicity before any test is run. Consider two explanations for why certain customers buy a boat. The first says that people older than 50 are the ones who buy boats. The second adds conditions about children and marital status on top of age. Both might fit the customers you already know, but the first rule is far more appealing. It uses less information, it is easier to check, and it is less likely to be tracking a coincidence in one particular group of customers. The second rule has more moving parts, and each extra part is another opportunity to capture something that is only true of the sample in front of you.
Overfitting: Fitting the Particulars
Overfitting is what happens when a model is too complex for its data, fitting the particularities of the training set so closely that it does not generalize. The model has spent its flexibility on noise, accidental patterns, and individual data points instead of on the structure that would carry over to new cases.
Think of a student who prepares for an exam by memorizing the answers to last year's questions word for word. On a test made of those exact questions, the student scores perfectly. Give the same student a question phrased slightly differently, or one covering the same idea from a new angle, and performance drops sharply. The student did not learn the underlying subject; they learned the particular examples. An overfitted model behaves the same way: it looks excellent on the examples it memorized and weak on anything phrased differently.
The danger of overfitting is that it hides. A model that has memorized its training set will produce a strong training number, and that number is exactly what makes the problem easy to miss. The strong figure on the training side can conceal a weak result on new data.
Underfitting: Missing the Structure
Underfitting is a model that is too simple to capture the variability in the data, so it performs poorly even on training data. It is the mirror image of overfitting. The problem is not that the model attended too closely to the training examples; it is that the model did not attend closely enough to the structure in them.
Imagine a rule stating that everybody who owns a house buys a boat. That rule is easy to state and easy to apply, but it ignores most of what actually distinguishes buyers from non-buyers. Because it misses the variation in the data, it does badly even on the examples used to build it. Since it fails on the training set, there is no point hoping it will rescue itself on new data.
This makes the training score especially informative in the underfitting case. Underfitting is usually visible right away: if a model cannot even reproduce the examples it was shown, it has not captured enough of the pattern to be useful anywhere.
The Complexity Dial and the Sweet Spot
Overfitting and underfitting are not two unrelated problems. They are two ends of one trade-off governed by model complexity. As the model is allowed to become more complex, it becomes better and better at predicting the training data, and that improvement never stops on the training side. On the new-data side the story is different. Up to a point, added complexity helps the model pick up real structure. Beyond that point, the model starts concentrating on each individual data point in the training set, and its ability to generalize falls.
Between these two regimes lies a sweet spot: the level of complexity that yields the best generalization performance. Finding that level is the real goal. A model that is too simple misses the structure, a model that is too complex memorizes the noise, and the target lies between them. The trade-off is usually illustrated by plotting training accuracy and test accuracy against model complexity. The training curve climbs steadily, while the test curve rises, peaks, and then falls as complexity keeps increasing.
Tracing a Model Across the Dial
Tracing the Boat Rules Across Complexity
Two rules try to explain which customers buy a boat. The simple rule uses only age, while the complex rule adds conditions about children and marital status. For each rule, trace what happens to training performance and to new-data performance.
- 1Start with the simple rule
The rule people older than 50 buy boats uses one piece of information. It is easy to check and does not chase coincidences in the sample. Its training performance may be modest, but it captures the main pattern.
- 2Add conditions to increase complexity
The complex rule adds children and marital status on top of age. Each added condition gives the rule another way to match the customers already known, which can push training performance upward.
- 3Check the training result
The complex rule now fits the known customers more closely, so its training score looks stronger than the simple rule's. This is the number that can hide the problem.
- 4Check the new-data result
Each extra condition may be tracking something true only of the sample in front of it. On customers the rule has never seen, the extra parts are more likely to mislead, so the complex rule can fall behind the simpler one.
Training performance rises with complexity, but new-data performance does not keep rising. The complex rule is the overfit pattern, and the simpler rule is closer to the balanced target, although the simplest possible rule can still underfit.
Reading the three columns together makes the trade-off concrete. The too-simple column is weak everywhere. The too-complex column looks impressive on the training side, which is exactly why it is dangerous, because the strong number hides the weak one. The sweet-spot column gives up a little training performance in exchange for the strongest result on new data. That exchange is what generalization means in practice.
Mistakes Beginners Make
Common mistake Judging a model by its training score alone
What it looks likeA model scores almost perfectly on the examples it was built from, so it is declared excellent.
Why it failsA strong training result is consistent with overfitting, and the real question is how the model behaves on cases it has never seen.
Do this insteadEvaluate on a held-out test set of examples kept back from training.
Common mistake Assuming more complexity always improves a model
What it looks likeAdding more conditions to a rule keeps raising its training performance, so the rule seems to keep getting better.
Why it failsTraining performance keeps improving, but new-data performance rises only up to a point and then falls.
Do this insteadLook for the sweet spot where generalization is strongest rather than the most complex model.
Common mistake Treating a poor training score as a problem to fix later
What it looks likeA model that misses many of its own training examples is kept on the theory that it will do better on new data.
Why it failsA model that fails on the training set has not captured enough of the pattern, so there is no reason to expect it to succeed on new data.
Do this insteadTreat weak training performance as a sign of underfitting and increase the model's ability to capture structure.
Common mistake Confusing simplicity with a guarantee of correctness
Predicting the Overfit Result
What do you think happens?
Two models are built from the same training examples. Model A is a simple rule that misses much of the pattern. Model B is so complex that it reproduces every training example. Which model shows the strongest training score, and which one is more likely to be overfitting?
Reveal answer
Model B has the strongest training score and is the one overfitting.
Overfitting spends the extra complexity on individual training points, so the training score looks strong while new-data performance suffers. Model A is the underfitting case, weak even on the training data.
A model's training score keeps climbing as you add complexity, but its performance on held-out examples has begun to drop. Explain in your own words why the training score cannot be used to decide when to stop adding complexity, and name the regime the model has entered.
Hints
- Think about what the training score measures and what the held-out examples measure.
- Recall the description of the regime where the model starts concentrating on individual data points.
Key Takeaways
- Generalization, meaning performance on new data, is the real measure of a model, and a strong training score alone can mislead.
- Overfitting happens when a model is too complex for its data, fitting training particularities and memorizing noise so it fails on new examples.
- Underfitting happens when a model is too simple to capture the variability in the data, so it performs poorly even on training data.
- Overfitting and underfitting sit at opposite ends of one complexity dial, and the target is the sweet spot that gives the best generalization performance.