Parameter Settings and Model Tuning
Model complexity describes how well a model can fit the patterns in its training data, and it is something that can be adjusted.
The Score That Looks Too Good
A common first instinct when building a model is to check how well it performs on the examples it was built from. If the score on those examples is high, it is tempting to conclude that the model is good. This article explains why that conclusion can be wrong, and where the real goal of a machine learning model lies.
What do you think happens?
A model scores very highly on the examples it was trained on. Is that enough to call it a good model?
Reveal answer
No. The training score alone does not show that the model will work on new cases.
Training performance is only a starting point. What matters is whether the patterns the model learned still hold when it meets cases it has never seen before.
What Model Complexity Means
Model complexity describes how well a model can fit the patterns in its training data, and it is something that can be adjusted.
Complexity is best understood as a dial rather than a yes or no property. A model with little capacity can only represent simple relationships in the training data. A model with greater capacity can follow more intricate variations. Complexity is therefore not good or bad in itself. What matters is whether the amount of complexity suits the problem and the data at hand.
Complexity is not a fixed feature you simply accept. Every algorithm covered in the source material offers ways to control model complexity, so you adjust it deliberately, and adjusting it wrongly is what leads to the two failure modes described later in this article.
Tracing Complexity Along the Dial
Imagine you start with a model that has very little capacity and then gradually raise its complexity. At the low end, the model cannot represent the variations in the training data, so it is too simple. As complexity increases, it becomes able to follow the training patterns more accurately. Past a certain point, however, it begins to follow the training data so closely that it stops describing the underlying pattern and starts describing the particular examples it saw. The diagram below traces this progression as three states.
Notice that the middle state is not simply the most complex model. It is the position where the amount of complexity suits the data. Moving right past that point does not add value; it only makes the model fit the training examples more tightly at the expense of new cases.
Training Performance Versus Generalization
Generalization is the ability to perform well on new, previously unseen data. It is the real goal of a model. Performance on the training examples is useful, but it is a different measurement. A model that memorizes the examples it was shown has achieved something, yet not the thing that is actually needed. The question that matters is whether the patterns it learned still hold when it meets cases it has never encountered.
| Aspect | Performance on training data | Performance on new data |
|---|---|---|
| What it measures | How well the model fits the examples it was built from | Whether the learned patterns hold for unseen cases |
| Role in evaluation | A starting point | The real goal of the model |
| Can it be misleading alone? | Yes, a high score may hide a model that memorized the examples | It is the measure that shows whether the model generalizes |
Two ways of evaluating a model, and why they differ
Two Ways Fitting Goes Wrong
Model complexity is closely tied to two ways a model can fail to generalize. Underfitting and overfitting sit at opposite ends of the complexity dial. Both describe a mismatch between the model and the training data, but in opposite directions. Each is examined below.
Locating Underfitting and Overfitting on the Dial
The diagram below places the three outcomes side by side across increasing complexity. The goal is the middle column, where the model's behavior on new data is what counts. Read it from left to right: as complexity grows, the model moves from missing the training patterns, to capturing them in a way that generalizes, to memorizing the particular examples.
Worked Example: Reading Two Scores Together
Diagnosing a model from its two evaluations
A model is evaluated on the examples it was trained on, and then on a set of new examples it has never seen. The training score is very high, but the score on new data is much lower. What does this pattern suggest about the model's complexity?
- 1Identify what each score measures
The training score shows how well the model fits the examples it was built from. The new-data score shows whether the learned patterns hold for unseen cases.
- 2Compare the two scores
A high training score paired with a much lower new-data score means the model fits the training examples well but does not carry that fit over to new cases.
- 3Relate the pattern to the complexity dial
This gap is the signature of overfitting. The model has likely moved past the suitable point on the dial, following the particular training examples too closely.
- 4Decide what to adjust
Because complexity is adjustable, the next step is to change the model's parameter settings to reduce complexity and then evaluate again on new data.
The gap between a high training score and a lower new-data score points to overfitting, so the model's complexity should be reduced and the model re-evaluated on new data.
Parameter Settings Before You Apply an Algorithm
The discussion that closes this chapter connects complexity to a practical matter: the settings you choose when you apply an algorithm. For many algorithms, choosing the right parameters is important for good performance. Parameter settings are one of the main levers for controlling complexity, so they connect directly to underfitting and overfitting. A setting that suits one problem may push a model into underfitting or overfitting on another.
Representation matters as well. Some algorithms are sensitive to how the input data is represented, and in particular to how the features are scaled. An algorithm that behaves well on one representation may behave poorly on another, even when the underlying data is identical. For this reason, an algorithm's assumptions must be understood before it is applied. Knowing what an algorithm assumes about its data tells you which settings are sensible and how the input should be prepared.
Tuning as a Repeated Loop
Because complexity is something you adjust, tuning a model is a repeated process rather than a single choice. You adjust the settings, train the model, evaluate it on new data, and then decide whether another adjustment is needed. The loop below shows these steps. Each pass uses the evaluation on new data to judge the model, not the score on the training examples alone.
Mistakes Beginners Make When Tuning
Common mistake Judging a model by its score on the training examples
What it looks likeA model scores almost perfectly on the data it was built from, so the builder stops evaluating it.
Why it failsGeneralization, performance on new previously unseen data, is the real goal. A high training score does not show that the learned patterns hold for new cases.
Do this insteadAlways evaluate the model on new data before concluding that it is good.
Common mistake Treating more complexity as always better
What it looks likeA builder keeps raising complexity because each increase improves the training score.
Why it failsComplexity is a dial, not a measure of quality. Past the point where it suits the data, more complexity leads to overfitting.
Do this insteadAsk whether the amount of complexity suits the problem and data, and check the result on new data.
Common mistake Applying an algorithm without understanding its assumptions
Why it failsParameter settings and feature representation affect performance, so an algorithm's assumptions must be understood before it is applied.
Do this insteadRead what the algorithm assumes and what its parameters control before choosing settings.
Common mistake Ignoring how input features are scaled when an algorithm is sensitive to representation
Practice: Diagnosing a Model
A model shows a low score on its training examples and also a low score on new data. Using the ideas in this article, describe which failure mode this pattern most likely indicates, and state which direction on the complexity dial you would move the model.
Hints
- Recall that underfitting occurs when a model is too simple to capture the variations in the training data.
- Consider whether the model is missing the training patterns in the first place, or following the examples too closely.
- Complexity is adjustable, so think about raising or lowering it, then re-evaluate on new data.
Key Takeaways
- 1Model complexity describes how well a model can fit the patterns in its training data, and it is something you can adjust like a dial.
- 2Generalization, performance on new unseen data, is the real goal, not performance on the training examples.
- 3Underfitting occurs when a model is too simple to capture the training variations, while overfitting occurs when it fits the training data too closely and fails to generalize.
- 4Parameter settings and feature representation affect performance, so an algorithm's assumptions must be understood before it is applied.
At a glance
- Model complexity describes how well a model can fit the patterns in its training data, and it can be adjusted like a dial.
- Generalization, performance on new unseen data, is the real goal rather than performance on training examples.
- Underfitting happens when a model is too simple to capture training variations; overfitting happens when it fits training data too closely and fails to generalize.
- Parameter settings and feature representation affect performance, so an algorithm's assumptions must be understood before it is applied.