Model Generalization
Overfitting comes from a model too complex for its data; it works on the training set but fails to generalize.
Why a Perfect Score Can Mislead
Imagine a company that wants to know which customers will buy a boat. A data scientist has records for a group of customers. Each record shows the customer's details and whether that customer bought a boat. Because the answers are already in the records, it is possible to write a rule that sorts those customers correctly, even perfectly.
The question is whether a rule like that is useful. A rule that is right about every customer we already know may still be wrong about the customers we have not met yet. Keep that tension in mind as you read.
What do you think happens?
If a rule gets every known customer right, what more would you need to know before trusting it for a new customer?
Training Data, New Data, and Generalization
The training set is the data we already have. We use it to build a model. The ability of a model to predict well on data it has never seen is called generalization.
We do not build a model in order to describe the training set, because we already know those answers. We build it so that it can make predictions for data it has never seen. This is why a model that scores perfectly on its training data does not, by that fact alone, deserve trust. Accuracy on the training set tells us how well the model memorized the examples it studied. It does not tell us how the model will behave on a new customer.
The source is firm on one point: the only real measure of whether an algorithm will perform well on new data is its evaluation on the test set. The test set is data held back from training, so the model has not seen it while it was being built. Performance on that held-back data is the honest check on generalization.
In the diagram, the training set only feeds the process of building the model. The test set enters later, when the finished model is evaluated. That late entry is what makes the evaluation an honest check: the model is judged on examples it did not learn from.
Two Opposite Ways a Model Fails
Both overfitting and underfitting produce a model that predicts badly, but they fail in opposite directions and show different symptoms. Each is described below.
Complexity as a Dial
Model complexity is the level of intricacy a model is allowed to have. It is closely tied to how many adjustable parameters the model contains. The source makes two observations that pull in opposite directions. First, the more complex a model is allowed to become, the better it can predict on the training data. Second, once a model becomes too complex, it begins to focus too heavily on each individual training point, and it stops generalizing to new data.
Think of complexity as a dial. Turning it up gives the model more freedom to bend around the training examples, which improves its training performance. Past a certain point, that same freedom lets the model bend around noise, and its performance on new data falls.
The source describes the resolution of the tension as a sweet spot between the two extremes. If the best training performance comes from the most complex model, and the best generalization comes from a model that is not too complex, then the same choice cannot maximize both. The intermediate model yields the best generalization performance, and it is the model the course asks you to aim for.
The three columns show why a model should not be ranked by its training score alone. The too-complex model has the strongest training result, but the gap between its training and new-data results is large. The too-simple model has a small gap only because both results are poor. The intermediate model is the one whose results hold up on new data.
Checking a Model Against Three Cases
Classifying a model by its two scores
A model is built to predict which students will pass a course. You are given its results on the training set and on new students it has never seen. Decide whether it is too simple, too complex, or in the sweet spot. Here the training results are poor and the new-student results are poor.
- 1Check the training result
The training result is poor, so the model fails even on the data it was built from.
- 2Match the symptom
Failing on the training data is the mark of underfitting, which the source describes as a model too simple to capture the pattern.
- 3Check the new-student result
The new-student result is also poor, which fits an underfitted model that disappoints everywhere.
- 4Classify
The model is too simple. Adding complexity, within limits, is the direction to investigate.
Too simple: underfitting.
Now take a different case. The model scores excellently on the training students but poorly on new students. The training score looks impressive, yet the new-student score reveals the problem. This is the signature of a model too complex for its data, and the correct conclusion is overfitting, not success. Trust the new-student result, because that is the measure of generalization.
Mistakes When Judging a Model
Common mistake Ranking a model by its training score alone
What it looks likeChoosing the model with the highest accuracy on the training set without checking anything else.
Why it failsAccuracy on the training set shows how well the model memorized the examples it studied. It does not show how the model will behave on a new customer.
Do this insteadJudge the model by its evaluation on the held-back test set.
Common mistake Assuming that more complexity always helps
What it looks likeAdding conditions to a rule until it matches every known record.
Why it failsComplexity improves training performance only up to a sweet spot. Past it, the model bends around noise and new-data performance falls.
Do this insteadLook for the simplest model that generalizes well, not the most detailed one.
Common mistake Treating a poor score as proof of a good, cautious model
What it looks likeBelieving a model is careful because it performs badly everywhere.
Why it failsA model that performs badly even on the training data is underfitting. Its failure is real, not a sign of restraint.
Do this insteadTreat poor training performance as a sign the model is too simple to capture the pattern.
Common mistake Confusing overfitting with a model that is merely accurate on the test set
Practice: Classify Three Models
For each of three models, decide whether it is too simple, too complex, or in the sweet spot. Model A has poor training results and poor new-data results. Model B has excellent training results and poor new-data results. Model C has good training results and good new-data results. For each one, say how it would be expected to perform on the training data and on new students.
Hints
- Ask first whether the model fails on the training data. A model that fails there is too simple.
- A large gap between excellent training results and poor new-data results points to a model that is too complex.
- The sweet spot is the model whose results hold up on new data, not the one with the highest training score.
Key Takeaways
- A model is valuable for how well it predicts new data, so a perfect score on the training set is not enough. Generalization is the ability to perform well on unseen data, and the test set held back from training is the honest check.
- Overfitting comes from a model that is too complex for its data. It looks excellent on the data it studied and disappoints on new data.
- Underfitting comes from a model too simple to capture the pattern. It performs badly even on the training set.
- Increasing complexity improves training performance only up to a sweet spot. The simplest model that generalizes well, which is usually an intermediate level, is the target.