Bias-Complexity Tradeoff
Model selection is the task of choosing an algorithm for a particular problem and setting its parameters.
The Decision Before the Model
When a practical problem arrives, the challenge is not always to write down one algorithm. Several algorithms may be reasonable candidates, and each algorithm may allow several parameter settings. The practical task is deciding which algorithm and which settings fit the problem. That combined decision is called model selection.
Model selection is the task of choosing an algorithm for a particular problem and setting its parameters.
Two Parts of One Choice
Model selection has two connected decisions. First, choose an algorithm that could address the particular problem. Second, decide how to set that algorithm's parameters. A selected model is therefore defined by both the algorithm choice and the parameter choices.
| Decision | Question it answers | Why it belongs to model selection |
|---|---|---|
| Algorithm choice | Which algorithm should be considered for this particular problem? | Several algorithms may be reasonable candidates. |
| Parameter setting | How should the chosen algorithm be configured? | Each algorithm may offer several parameter settings. |
| Combined model choice | Which algorithm and settings fit the problem? | The selected model is defined by both decisions. |
Tracing AdaBoost's T
AdaBoost makes the role of parameter setting concrete. Its parameter T controls the bias-complexity tradeoff. Therefore, deciding how to set T is not a minor implementation detail that happens after model selection. It is part of model selection itself.
Treating T as a Model-Selection Decision
A practical problem can be addressed with AdaBoost. What decisions must be included when selecting the model?
Choose the algorithm: AdaBoost is one algorithm being considered for the particular problem.
Choose T: T is a parameter of AdaBoost, so its setting must also be decided.
Consider the tradeoff: Because T controls the bias-complexity tradeoff, the choice of T is connected to the relationship between the model's bias and its complexity.
Classify the decision: The algorithm choice and the T choice together form part of the model-selection problem.
Selecting AdaBoost without deciding how to set T leaves part of the model-selection decision unfinished.
Understanding the Tradeoff
The bias-complexity tradeoff is a consideration that arises when parameter settings control the relationship between a model's bias and its complexity. In the source's concrete example, AdaBoost's T controls this tradeoff. The important point is not that one value of T is always best. The appropriate choice is tied to the particular practical problem.
Avoiding False Shortcuts
Treating model selection as only an algorithm choice.
Model selection combines algorithm choice with parameter setting.
Fix:
Record both the algorithm being considered and the parameter values that configure it.Treating parameters as an afterthought.
Parameters are part of the model-selection decision, not an afterthought.
Fix:
Include parameter setting in the model-selection question from the beginning.Claiming that one algorithm or one parameter setting is universally best.
The selection question is tied to the particular practical problem at hand.
Fix:
Treat algorithm and parameter choices as decisions that must fit the problem.Calling the bias-complexity tradeoff an algorithm.
The tradeoff is a consideration related to parameter settings.
Fix:
Use the tradeoff to describe what parameter selection must consider.
Practice the Decision
A team is considering several algorithms for a practical problem. One candidate is AdaBoost. Explain, in your own words, why the team has not completed model selection merely by naming AdaBoost. Then explain why choosing T should be discussed as part of the same decision.
Hints
- Start with the two decisions that define model selection.
- Connect T to the bias-complexity tradeoff.
- Do not claim that one value of T is universally best.
Checking a Model-Selection Explanation
Which explanation is complete: “Model selection means choosing AdaBoost,” or “Model selection means choosing an algorithm and setting its parameters for a particular problem”?
Inspect the first explanation: It names an algorithm but leaves out parameter setting.
Inspect the second explanation: It includes both algorithm choice and parameter choices.
Apply the AdaBoost example: Because T controls the bias-complexity tradeoff, setting T belongs in the second explanation.
The second explanation is complete.
Key Takeaways
- Model selection is choosing an algorithm for a particular problem and setting its parameters.
- A selected model is defined by both the algorithm choice and the parameter choices.
- AdaBoost's parameter T controls the bias-complexity tradeoff.
- Choosing T is part of model selection, not merely a minor implementation detail.
- The bias-complexity tradeoff is a model-selection consideration rather than a separate algorithm.
Key Takeaways
- Model selection combines choosing an algorithm with setting its parameters.
- The selected model depends on both decisions.
- AdaBoost's T demonstrates why parameter setting matters: T controls the bias-complexity tradeoff.
- There is no universally best parameter setting stated here; the choice is tied to the practical problem.
- Bias-complexity tradeoff describes a consideration within model selection, not a separate algorithm.