Understanding Model Generalization
Generalization is the ability of a model to perform well on new, previously unseen data, and it is the central goal of building a model.
Why Training Results Are Not Enough
Imagine a student who memorizes the answers to last year's exam questions. On a practice test built from those same questions, the student scores perfectly. On the real exam, which contains questions the student has never seen in this exact form, the score drops sharply. The memorized answers looked like understanding, but they did not carry over to new material.
Machine learning faces the same problem. A model is built by showing it examples, and it is easy to build a model that looks excellent on those examples. The harder and more important question is whether the model still works when it meets data it has never encountered.
What do you think happens?
A model scores perfectly on the examples it was built from. Does that prove it will work well on new data?
Reveal answer
No. Perfect scores on the training examples do not by themselves show that the model will work on new data.
Generalization is about unseen data, so a model cannot be judged only by how it performs on the data used to build it. Two models can show identical results on training examples and still differ greatly on new inputs.
What Generalization Means
Generalization is the ability of a model to perform well on new, previously unseen data. It is the central goal of building a model.
The definition has two parts that matter. The first is the word new: the model is judged on inputs it did not learn from. The second is the word perform: it is not enough for the model to exist or to fit something. It has to produce useful results on that new material. This is why the chapter treats the ability to handle new data as the thing that matters most.
Training Data Versus Unseen Data
To check generalization, the available examples are divided into two portions. The model learns from the training portion. The other portion is held back, so the model never sees it during learning. The model is then evaluated on that held-back portion. The evaluation tells us how the model handles data that was not part of its learning, which is the situation generalization is about.
Follow the diagram from left to right. Every example starts in the same pool, but only the training portion influences what the model learns. The held-back portion acts as the stand-in for new data. If the model does well only on the portion it learned from, the evaluation on the held-back portion will reveal that gap.
Model Complexity as the Lever
Model complexity describes how elaborate a model is and how much it can adjust to the data. Complexity matters because it directly influences generalization. It is closely tied to generalization and it shapes which failure pattern a model is likely to show.
A useful way to think about complexity is as a lever with a range of settings. At one end, the model is too simple for the problem. At the other end, the model is too complex in the way it fits the training data. Between these two ends is a setting matched to the problem and to the data.
Complexity is not good or bad on its own. What matters is whether it is matched to the problem and to the data.
The Two Failure Patterns
The source presents two classic failure patterns, underfitting and overfitting, as the two ends of the complexity spectrum. Each one is defined by what the model does with the training data. The fitted curves below show how the two differ in shape.
Spotting the Pattern in a Scenario
The clearest way to diagnose a model is to compare how it performs on the examples it learned from with how it performs on new examples. The difference between those two results is one useful way to read the situation. The table below uses illustrative numbers for three models trained on the same task. They are not drawn from the source and are included only to show the reasoning.
| Model | Score on training examples | Score on new data | Difference |
|---|---|---|---|
| Model A | 55% | 52% | 3 points |
| Model B | 99% | 61% | 38 points |
| Model C | 90% | 88% | 2 points |
Illustrative scores for three models on the same task
Reading the three models
Which of the three models shows the signs of underfitting, which shows the signs of overfitting, and which generalizes?
- 1Check Model A
Model A scores low on the training examples and also low on new data. It has not captured the pattern even in the data it learned from, which matches the signs of underfitting. It is too simple for the problem.
- 2Check Model B
Model B scores very high on the training examples but drops sharply on new data. The large difference shows that it has fit the training examples closely without carrying that fit over to unseen inputs. This matches overfitting, where the model is too complex in the way it fits the training data.
- 3Check Model C
Model C scores high on both the training examples and the new data, and the two results are close. This is the pattern of a model whose complexity is matched to the problem and the data.
Model A shows underfitting, Model B shows overfitting, and Model C generalizes. The pattern matters more than any exact cutoff: the important signal is low results on both sides for underfitting and high training results paired with much lower new-data results for overfitting.
Mistakes Beginners Make About Generalization
Common mistake Judging a model only by its results on the data it was built from
What it looks likeA model scores perfectly on its training examples, so it is declared ready to use.
Why it failsGeneralization is about new, unseen data. Two models can match on training examples and still differ greatly on new inputs.
Do this insteadCheck performance on data the model did not learn from before drawing any conclusion.
Common mistake Assuming a more complex model is always better
What it looks likeA very elaborate model is chosen because it fits the training points more closely than a simpler one.
Why it failsComplexity is not good or bad on its own. Too much complexity can lead to overfitting, where the model follows the training data too closely.
Do this insteadMatch the complexity to the problem and the data, and judge the result on new data.
Common mistake Assuming a simple model is always the safe choice
What it looks likeA very simple model is kept because it is easy to understand, even though it misses the pattern in the training data.
Why it failsA model that is too simple cannot capture the variations present in the training data, which is underfitting.
Do this insteadLook for a setting between the two ends of the spectrum where the model handles both the training data and new data well.
Common mistake Treating a single result as proof, instead of comparing performance on training examples with performance on new examples
Classify a Scenario
A model scores 58% on its training examples and 56% on new data. Which failure pattern does this result most resemble, underfitting or overfitting? Explain your answer using the idea of complexity.
Hints
- Ask whether the model captures the pattern even in the data it learned from.
- Compare how far apart the two scores are, and whether both are low or only the training score is high.
- Think about whether the model is more likely to be too simple or too complex for the problem.
Key Takeaways
- Generalization is the ability of a model to perform well on new, previously unseen data, and it is the central goal of building a model.
- Good performance on the examples a model was built from does not by itself show that the model will work on new data.
- Model complexity is closely tied to generalization. Underfitting occurs when a model is too simple and cannot capture the variations in the training data, while overfitting occurs when a model is too complex and focuses too much on the training data.
- Complexity is not good or bad on its own; what matters is whether it is matched to the problem and to the data.
- To diagnose a model, compare its results on training examples with its results on new data, and look for low performance on both sides (underfitting) or high training performance paired with much lower new-data performance (overfitting).