Model Complexity and Parameters
Model complexity is the level of intricacy a model is allowed to have, and it is closely tied to how many adjustable parameters the model contains.
A Rule That Knows Every Customer
Imagine a company that wants to know which customers will buy a boat. A data scientist has records for a group of customers. Each record shows the customer's details, along with whether that customer bought a boat. Because the answers are already in the records, it is possible to write a rule that sorts those customers correctly, even perfectly.
The question this article takes up is whether a rule like that is useful. A rule that is right about every customer we already know may still be wrong about the customers we have not met yet. Keep that tension in mind as you read.
What do you think happens?
If a rule gets every known customer right, what more would you need to know before trusting it for a new customer?
Reveal answer
You would need to know how the rule performs on customers who were not part of the records used to build it.
A perfect score on known customers only shows how well the rule fits those records. The honest test is how it behaves on data it has never seen, which the source calls the test set.
Training Data Versus New Data
The training set is the data we already have. We use it to build a model. We do not build a model in order to describe the training set, because we already know those answers. We build it so that it can make predictions for data it has never seen. The ability to do that well on unseen data is called generalization.
This is why a model that scores perfectly on its training data does not, by that fact alone, deserve trust. Accuracy on the training set tells us how well the model memorized the examples it studied. It does not tell us how it will behave on a new customer.
The test set is data held back from training, so the model has not seen it while it was being built. Performance on that held-back data is the honest check on generalization. The source is firm that the only real measure of whether an algorithm will perform well on new data is its evaluation on the test set.
Turning the Complexity Dial
Model complexity is the level of intricacy a model is allowed to have, and it is closely tied to how many adjustable parameters the model contains. Parameters are the adjustable settings that let a model bend its shape around the examples it is given. More parameters, and more freedom in how they are set, generally means a more complex model.
Think of complexity as a dial. Turning it up gives the model more freedom to bend around the training examples, which improves its training performance. Past a certain point, that same freedom lets the model bend around noise, and its performance on new data falls. The diagram below traces this progression as the dial moves, so you can watch the two kinds of performance separate.
Read the diagram from left to right. The training performance keeps improving as complexity grows, but the new-data performance does not follow it all the way. The two measures move together at first and then separate. That separation is the central idea of this article.
Overfitting and Underfitting Compared
The two failures can be held side by side. Both produce a model that predicts badly, but they fail in opposite directions and show different symptoms. The comparison below sets a too-simple model, a well-fitted model, and a too-complex model against the same three questions.
The comparison shows that a good training score alone cannot distinguish a well-fitted model from an overfitted one. Both can look strong on the examples they studied. Only the behavior on new data separates them.
Finding the Sweet Spot
If the best training performance comes from the most complex model, and the best generalization comes from a model that is not too complex, then the two goals cannot both be maximized by the same choice. The source describes the resolution as a sweet spot somewhere between the two extremes. That intermediate model is the one that yields the best generalization performance, and it is the model the course asks you to aim for.
Checking Three Customer Models
Classifying three candidate rules for the boat buyers
Three candidate rules are proposed for predicting boat purchases. Rule A is simply 'people older than 50 want to buy a boat'. Rule B adds conditions about children and marital status to age. Rule C contains many conditions that match every customer in the records one by one. For each rule, decide whether it is too simple, too complex, or in the sweet spot, and predict how it will perform on training data and on new customers.
- 1Check Rule A against the training data
Suppose Rule A explained the behavior of every customer in the data. The source treats this as a case where the simple rule is easier to trust than a longer rule. Its training performance is good, and because it is short, it is not built from accidental features of the records.
- 2Compare Rule B to Rule A
Rule B adds children and marital status to age. The source says a longer rule like this would be harder to trust than the simple one that already explains the data. If Rule B adds only conditions that help it describe the real pattern, it sits near the sweet spot. If it adds conditions that only fit the records, it begins to overfit.
- 3Check Rule C for overfitting
Rule C matches every customer through many conditions. The source warns that a model with many conditions can be shaped to fit almost any collection of examples, including accidental features. Its training score is excellent, but this is the warning sign: it is too complex for the information available and is expected to disappoint on new customers.
- 4Predict performance on new customers
Rule A is expected to generalize reasonably if the pattern is real, Rule B may generalize best if it stays near the sweet spot, and Rule C is expected to perform worse on new customers than on the records it was built from.
Rule C is the overfitted candidate despite its perfect training score. The simplest rule that describes the real pattern is the target, and a model should be judged on how it performs on the test set rather than on training accuracy.
Mistakes Beginners Make With Complexity
Common mistake Judging a model by its training score
What it looks likeA model gets every known customer right, so the analyst deploys it immediately.
Why it failsAccuracy on the training set shows how well the model memorized the examples it studied. It does not show how it will behave on a new customer.
Do this insteadEvaluate the model on the held-back test set, which the model did not see during building.
Common mistake Assuming more complexity always helps
What it looks likeAdding more conditions to a rule until it matches every record.
Why it failsComplexity improves training performance only up to a sweet spot. Past that point, the model bends around noise and new-data performance falls.
Do this insteadIncrease complexity only while it improves generalization, and stop at the simplest model that does the job.
Common mistake Treating underfitting as the safe choice
What it looks likeChoosing a very simple rule because it looks trustworthy, even though it misses the pattern.
Why it failsAn underfitted model performs badly even on the training set, so it has no hope of describing new data either.
Do this insteadCheck whether the simple model fails on the training data too. If it does, add complexity until it captures the pattern.
Common mistake Confusing the number of conditions with the quality of the pattern found
Preferring the Simplest Adequate Model
There is a practical preference that comes from experience. Simple models are expected to generalize better to new data than complicated ones. For this reason, the guiding aim is to find the simplest model that does the job. A short rule that explains the data is easier to trust than a longer rule that adds conditions only to match the records more closely.
When comparing candidate models, record both the training performance and the performance on the held-back test set. Prefer the simplest model whose test performance is near the best, and be suspicious of any model whose training score is high while its test score is weaker.
Practice: Judging Three Cases
Take three models built on the same records. Model X scores poorly on the training set. Model Y scores well on the training set and well on the test set. Model Z scores perfectly on the training set but poorly on the test set. For each model, decide whether it is underfitting, in the sweet spot, or overfitting, and explain how it would be expected to perform on new customers.
Hints
- Start with the training score. A model that fails on the training set is too simple for the pattern.
- A model that does well on the training set but poorly on held-back data is too complex for its data.
- The sweet spot is the model that generalizes best, which is judged on the test set rather than on training accuracy.
Summary of the Complexity Trade-Off
- 1A model must perform well on new data, because the purpose of building it is to predict unseen cases, a skill called generalization.
- 2Overfitting comes from a model too complex for its data: it works on the training set but fails to generalize.
- 3Underfitting comes from a model too simple to capture the pattern: it performs badly even on the training set.
- 4Complexity improves training performance only up to a sweet spot, and the simplest model that generalizes well is the target.
At a glance
- A model must perform well on new data, because its purpose is to predict cases it has never seen. A perfect score on the training set is not enough.
- Overfitting comes from a model too complex for its data, which works on the training set but fails to generalize.
- Underfitting comes from a model too simple to capture the pattern, and it performs badly even on the training set.
- Increasing complexity improves training performance, but past a sweet spot it hurts generalization, so the simplest model that generalizes well is the target.