Validation Techniques

A model-selection curve compares training and validation error across model complexity.

Programming

Why Complexity Needs Validation

A more complex model does not automatically produce a better model. To study the effect of complexity, compare how training error and validation error change across models with different levels of complexity. This comparison is called a model-selection curve. It helps reveal when increasing complexity is associated with overfitting.

The central question is not simply whether a model becomes more complex. The important question is whether validation error continues to improve as complexity increases.

Reading the Model-Selection Curve

A model-selection curve compares two error measures across model complexity. Training error measures the model's error as it is fit with increasing complexity, while validation error shows how the model performs on validation data. Reading the two curves together is essential because they follow different trends.

How do training error and validation error change together as model complexity increases?

The visual pattern has two parts. Training error decreases as complexity increases. Validation error first decreases, but after complexity continues to increase, validation error starts to increase. That contrast is the key evidence used to diagnose overfitting.

Following Training Error

The training-error trend is monotonic in this comparison: as model complexity increases, training error decreases. A more complex model can therefore look continuously better when judged only by its training error.

What happens to training error as the model becomes more complex?

Finding the Validation Minimum

Validation error behaves differently from training error. As complexity begins to increase, validation error first decreases. If complexity continues to increase, validation error eventually starts to increase. The lowest point in this pattern marks the point before the later worsening begins.

What happens to validation error as model complexity increases, and where does it reach its minimum?

Comparing Three Complexity Levels

A model-selection comparison includes a lower-complexity model, a more complex model, and a still more complex model. The training error decreases at each step. The validation error decreases from the first model to the second, then increases for the third. What does the comparison show?

Compare training error: Training error decreases as complexity increases, so the third model appears better if training error is the only measure considered.

Compare validation error: Validation error improves from the first model to the second, but worsens for the third model.

Interpret the contrast: The later increase in validation error, despite the continuing decrease in training error, is the pattern that signals overfitting.

The comparison identifies the later, more complex model as showing the overfitting pattern. The validation curve must be interpreted together with the training curve.

Recognizing Overfitting

Overfitting is signaled when training error continues to decrease while validation error starts to increase as model complexity increases. The model is improving according to training error but worsening according to validation error. The evidence is the rising validation error, not complexity by itself.

What pattern in the training and validation error curves shows that the model has started fitting noise rather than generalizing?

The diagnostic sequence is: training error decreases, validation error first decreases, and then validation error increases while training error continues to decrease. That final change in validation error is the signal to identify.

How does increasing model complexity move a model from higher error toward a validation minimum and then toward overfitting?

Common Reading Mistakes

Choosing the most complex model because it has the lowest training error.
Training error alone does not reveal the later increase in validation error.
Fix: Compare training error and validation error together.
Assuming that a complex model is automatically better.
A more complex model can be associated with increasing validation error.
Fix: Look for the point where validation error first decreases and then begins to increase.
Calling a model overfit only because it is complex.
The source identifies the increase in validation error as the evidence of overfitting.
Fix: Diagnose overfitting from the pattern of rising validation error as complexity increases.
Ignoring the validation curve after finding that training error is lower.
The two curves must be interpreted together.
Fix: Use the validation trend to check whether additional complexity is associated with a worsening result.

Practice Diagnosis

EASY

A model-selection curve shows that training error decreases steadily as model complexity increases. Validation error decreases at first, reaches its lowest point, and then increases. State the diagnosis and identify the evidence that supports it.

Hints

Compare the direction of the training-error trend with the direction of the validation-error trend after the validation minimum.
The diagnosis depends on the increase in validation error, not merely on the model being complex.

Your attempt (not saved)

What do you think happens?

If training error continues to decrease but validation error begins to increase as complexity rises, what pattern should you identify?

The model-selection curve shows overfitting
The model is automatically improving
Training and validation error are following the same trend

Reveal answer

Answer: The model-selection curve shows overfitting.

The defining evidence is that validation error increases while training error continues to decrease.

Key Takeaways

A model-selection curve compares training error and validation error across different levels of model complexity.
Training error decreases as model complexity increases.
Validation error first decreases and then increases as complexity continues to increase.
The increase in validation error is the key signal of overfitting.
Training and validation error must be interpreted together; training error alone can make increasing complexity look continuously beneficial.

Key Takeaways

A model-selection curve compares training and validation error across model complexity.
Training error decreases as complexity increases.
Validation error first decreases, then increases.
Overfitting is indicated when validation error rises while training error continues to fall.