Risk and Prediction Error
Approximation error is a limitation in what the model can capture.
A Poor Result Has More Than One Cause
Suppose a learning algorithm produces a poor predictor. The poor result by itself does not tell you why the predictor failed. The model may be too limited to represent the underlying relationship, or it may be too complex and too closely tied to noise in the data. These possibilities lead to different diagnoses: approximation error concerns the model's limitations, while estimation error concerns how accurately the model's true parameters were estimated. Model selection must balance these two kinds of error.
Start with the Model's Limitation
A model-selection problem involves deciding how suitable a model is for representing the relationship being studied. If the available model is too restricted, some part of the underlying relationship cannot be captured, even before we consider how well its parameters were estimated. That limitation is approximation error. The model is making a systematic compromise because its form cannot express everything that matters.
Approximation error is a limitation in what the model can capture.
Approximation Error in a Model Choice
A model that is too limited
Imagine comparing two candidate models for a relationship whose pattern is more complicated than a simple straight trend. One candidate model can express only a simple trend.
Inspect the model form: The candidate model has limited expressive ability. It cannot represent every part of the underlying relationship.
Identify the limitation: The portion of the relationship that the model cannot capture is the model's approximation error.
Name the likely diagnosis: If the model is too simple for the relationship, the associated problem is underfitting.
Approximation error comes from what the selected model is unable to represent, not from a failure to estimate its parameters accurately.
This example separates model limitation from parameter estimation. Even a more accurate parameter estimate cannot make a model express patterns that its form does not allow. In model-selection language, a model that is too simple is associated with underfitting.
Estimation Error from Finite Evidence
Estimation error is a limitation in accurately estimating the model's true parameters. Here the issue is not primarily what the model form can represent. The issue is how accurately the parameter values of that model have been determined. A model may be capable of representing a useful relationship while the fitted parameter values remain inaccurate.
A capable model with inaccurate parameters
Imagine that a chosen model has enough expressive ability for the underlying relationship, but the parameter values fitted for it are inaccurate.
Check the model's capability: The model form is not necessarily too limited. It can represent the relevant relationship.
Check the fitted values: The estimated parameter values do not accurately match the model's true parameters.
Name the error: The mismatch is estimation error because the limitation concerns accurate parameter estimation.
A model can have low limitation from its form while still suffering from estimation error.
The Complexity Trade-off
Changing model complexity changes the balance that model selection must manage. A model that is too simple is associated with underfitting and may leave important structure uncaptured. A model that is too complex is associated with overfitting and fits noise. The total prediction risk is therefore not diagnosed by asking only whether the model is simple or complex. The useful goal is to balance approximation error against estimation error rather than optimizing one in isolation.
Underfitting and Overfitting
Underfitting and overfitting are two different explanations for a poor predictor. Underfitting is associated with a model that is too simple, so the model is too limited to capture the underlying relationship. Overfitting is associated with a model that is too complex and fits noise in the data. Both can produce poor results, but the model limitations are opposite.
| Diagnosis | Model limitation | Relevant error |
|---|---|---|
| Underfitting | The model is too simple | Approximation error |
| Overfitting | The model is too complex and fits noise | Estimation error is part of the balancing concern |
Diagnose Before You Select
- First ask whether the model is too limited to represent the underlying relationship.
- If it is too limited, identify approximation error and consider the underfitting diagnosis.
- Then ask whether the model is too complex and too closely tied to noise in the data.
- If it fits noise, identify the overfitting diagnosis and consider estimation error as part of the trade-off.
- Choose a model by balancing approximation error and estimation error rather than focusing on only one of them.
Treating every poor predictor as evidence that the model is too simple.
The same observed outcome can have different causes.
Fix:
Inspect whether the model is limited in what it can capture or overly tied to noise.Treating approximation error and estimation error as the same problem.
One is a limitation of the model's expressive capacity, while the other is a limitation in parameter estimation.
Fix:
Name the limitation before deciding how to change the model.Choosing the most complex model automatically.
Complexity can produce overfitting rather than a better predictor.
Fix:
Balance the benefit of representing more of the relationship against the risk of fitting noise.
Check Your Diagnosis
A learning algorithm produces a poor predictor. You discover that the chosen model is too simple to represent an important part of the underlying relationship. Which concept best describes the limitation, and which fitting problem is associated with it?
Hints
- Focus on what the model can capture before thinking about parameter accuracy.
- The associated fitting problem is linked to a model that is too simple.
What do you think happens?
What is the diagnosis?
Reveal answer
Answer: Approximation error associated with underfitting
The model is too limited to represent the underlying relationship, which is the defining limitation of approximation error in this context. A model that is too simple is associated with underfitting.
Now consider the opposite diagnosis: the model is very complex and closely follows noise in the data. Explain why this is not simply another example of underfitting, and identify the model-selection concern it creates.
Hints
- Compare the model's complexity with the definition of underfitting.
- Use the distinction between a model that is too limited and one that fits noise.
Key Takeaways
- Approximation error is the limitation in what a model can capture.
- Estimation error is the limitation in accurately estimating the model's true parameters.
- Underfitting is associated with a model that is too simple.
- Overfitting is associated with a model that is too complex and fits noise.
- Model selection requires balancing approximation error and estimation error instead of assuming that every poor predictor has the same cause.
Key Takeaways
- Approximation error describes what the model cannot represent.
- Estimation error describes inaccurate estimation of the model's true parameters.
- Underfitting comes from a model that is too simple, while overfitting comes from a model that is too complex and fits noise.
- A poor predictor requires diagnosis: the cause may be limited representation or excessive dependence on noise.
- Good model selection balances approximation and estimation errors.