Evaluating Models on a Test Set
A model is valuable for how well it predicts new data, so a perfect score on the training set is not enough.
A Rule That Is Right About Everyone We Know
Imagine a company that wants to know which customers will buy a boat. A data scientist has records for a group of customers. Each record shows the customer's details along with whether that customer bought a boat. Because the answers are already in the records, it is possible to write a rule that sorts those customers correctly, even perfectly. The real question is whether such a rule is useful. A rule that is right about every customer we already know may still be wrong about the customers we have not met yet.
What do you think happens?
If a rule gets every known customer right, what more would you need to know before trusting it for a new customer?
Reveal answer
You would need to know how the rule performs on customers it has never seen, because a rule that is right about known customers may still be wrong about new ones.
The known customers are the training data. Their answers are already in the records, so a perfect score on them says how well the rule memorized them, not how it will behave on someone new.
Keep this tension in mind as you read the rest of the article. The goal of evaluation is to find out whether a model can be trusted on the data it was never shown, and the sections below trace how that trust is earned.
Tracing a Rule Through the Boat Scenario
Two candidate rules can both explain the known customers. The first is simple: people older than 50 want to buy a boat. The second is longer: it uses age and also adds children and marital status. Follow each rule through the same steps to see what the data can and cannot tell you.
Choosing between a simple rule and a longer rule
Two rules both describe the behavior of every customer in the training data. The simple rule uses age alone. The longer rule adds children and marital status to age. Which one should be preferred, and why?
- 1Check the simple rule against the training records
The simple rule, that people older than 50 want to buy a boat, explains the behavior of every customer in the data.
- 2Check the longer rule against the same records
The longer rule also explains every customer, but it needs more conditions to do so: age, children, and marital status.
- 3Apply the preference for simplicity
Both rules fit the known customers, but the simple rule is easier to trust. The guiding aim is to find the simplest model that does the job.
Both rules fit the known customers equally well on the training data, but the simpler rule is the one to prefer. Fitting the known data is not the deciding factor; the simpler rule is more likely to hold up on new customers.
Training Data Versus Unseen Data
The training set is the data we already have. We use it to build a model. We do not build a model in order to describe the training set, because we already know those answers.
Generalization is the ability of a model to predict well on data it has never seen. A model is valuable for how well it predicts new data, so generalization is the real goal.
The test set is data held back from training, so the model has not seen it while it was being built. The source is firm on one point: the only real measure of whether an algorithm will perform well on new data is its evaluation on the test set. Performance on that held-back data is the honest check on generalization. The diagram below shows how a dataset moves from being split, to training, to being evaluated on the held-back portion.
Model Complexity as a Dial
Model complexity is the level of intricacy a model is allowed to have. It is closely tied to how many adjustable parameters the model contains. Think of complexity as a dial. Turning it up gives the model more freedom to bend around the training examples, and that improves its performance on the training data.
The source makes two observations that pull in opposite directions. First, the more complex a model is allowed to become, the better it can predict on the training data. Second, once a model becomes too complex, it begins to focus too heavily on each individual training point, and it stops generalizing to new data. Past a certain point, the same freedom that let the model fit the examples now lets it bend around noise, and its performance on new data falls.
If the best training performance comes from the most complex model, and the best generalization comes from a model that is not too complex, then the two goals cannot both be maximized by the same choice. The resolution the source describes is a sweet spot somewhere between the two extremes. That intermediate model yields the best generalization performance, and it is the model the course asks you to aim for. A model should therefore not be ranked by its training score alone, because pushing that score higher by adding complexity can carry you past the sweet spot and into overfitting.
Overfitting and Underfitting Side by Side
Both overfitting and underfitting produce a model that predicts badly, but they fail in opposite directions and show different symptoms. The comparison below sets them against each other on the training data and on new data.
Tracing Three Models Through the Same Scenario
The source suggests a way to check your reasoning: for each case, ask whether the model is too simple, too complex, or in the sweet spot, and then ask how it would be expected to perform on the training data and on new students. The illustration below applies that check to three models that predict whether a student will pass a course.
Three models, three expected outcomes
Three models have been built from the same small set of student records. Model A uses one condition. Model B uses a few sensible conditions. Model C uses many specific conditions that match nearly every record. For each model, decide whether it is too simple, too complex, or in the sweet spot, and predict its performance on the training data and on new students.
- 1Model A: one condition
Model A is too simple for the pattern. It is expected to perform badly on the training data, and therefore badly on new students as well.
- 2Model B: a few sensible conditions
Model B is in the sweet spot. It captures the pattern without chasing accidental features, so it is expected to perform well on the training data and to generalize well to new students.
- 3Model C: many specific conditions
Model C is too complex for the amount of information it was built from. It is expected to score almost perfectly on the training data, but to disappoint on new students because it has learned accidental features of the records.
Model B is the target. Model C shows that a near-perfect training score can hide overfitting, and Model A shows that a simple model can fail everywhere, including the data it studied.
Mistakes When Judging a Model
Common mistake Ranking a model by its training score alone
What it looks likeA model scores perfectly on every record it was built from, so it is declared the best model.
Why it failsA training score tells us how well the model memorized the examples it studied. It does not show how the model behaves on a new customer or a new student.
Do this insteadJudge the model by its performance on the held-back test set, which is the honest check on generalization.
Common mistake Assuming more complexity always means a better model
What it looks likeA model with many conditions is chosen over a simpler one because it fits the known records more closely.
Why it failsComplexity improves training performance only up to a sweet spot. Past that point the model starts to focus on individual training points and stops generalizing.
Do this insteadLook for the simplest model that does the job, and check whether its performance holds on data it has not seen.
Common mistake Dismissing an underfitted model because it looks clever
What it looks likeA very simple rule is rejected because it seems too basic to be useful.
Why it failsAn underfitted model fails even on the training data, so its weakness shows up immediately and it cannot be trusted on new data either.
Do this insteadCheck whether the model performs badly on the training data. If it does, the model is too simple and needs more capacity to capture the pattern.
Common mistake Confusing a sweet spot with the most complex model that can be built
Checking Your Reasoning on a New Model
A new model predicts boat purchases. It scores perfectly on the training records and performs badly on the held-back test set. Is it overfitting, underfitting, or in the sweet spot? Explain what you would change, and why the training score alone cannot settle the question.
Hints
- Ask first how the model performs on the training data and then how it performs on the held-back test set.
- A near-perfect training score combined with poor test performance is the pattern of a model that is too complex for its data.
- Consider whether reducing the complexity could help the model generalize, while still capturing the pattern.
Key Takeaways
- A model is valuable for how well it predicts new data, so a perfect score on the training set is not enough.
- The test set is data held back from training, and performance on it is the honest check on generalization.
- Overfitting comes from a model too complex for its data and fails to generalize; underfitting comes from a model too simple to capture the pattern and performs badly even on the training set.
- Complexity improves training performance only up to a sweet spot, and the simplest model that generalizes well is the target.