Feature Selection and Model Design
Selecting features determines the dimensionality of the feature space.
Choosing the Inputs
When designing a learning algorithm, choosing the algorithm itself is only part of the decision. You must also decide which available patient features the algorithm will use. In the heart-attack prediction example, the available features are blood pressure, body-mass index, age, level of physical activity, and income. Selecting a subset of these features determines the dimensionality of the feature space.
Feature selection is also a model-design decision: changing the selected features changes the size of the model's input space.
Two Input Paths
Two possible heart-attack prediction designs
Compare an algorithm that uses blood pressure and body-mass index with an algorithm that uses blood pressure, body-mass index, age, physical activity, and income.
Select two features: The first design uses blood pressure and body-mass index. Its feature space is two-dimensional.
Select five features: The second design uses blood pressure, body-mass index, age, physical activity, and income. Its feature space is five-dimensional.
Compare the designs: The first design works in a smaller feature space. The second works in a larger feature space, so the comparison becomes a question about the advantages and demands associated with different levels of complexity.
The two algorithms do not receive the same feature set: one receives two selected features, while the other receives all five listed features.
The diagram shows the central design difference. Both models can be used for the same prediction task, but the two-feature algorithm receives only blood pressure and body-mass index. The five-feature algorithm receives those two measurements plus age, physical activity, and income. Because their inputs differ, their feature spaces differ as well.
Dimensionality in the Feature Space
Dimensionality here means the number of selected features used as coordinates in the feature space. Selecting blood pressure and body-mass index creates a two-dimensional feature space. Adding age, physical activity, and income creates a five-dimensional feature space. The dimensionality therefore changes directly with the number of selected features.
| Feature selection | Selected features | Feature-space dimensionality |
|---|---|---|
| Smaller selection | Blood pressure and body-mass index | 2D |
| Larger selection | Blood pressure, body-mass index, age, physical activity, and income | 5D |
The number of selected features determines the dimensionality in this heart-attack prediction example.
Complexity and Bias
Feature selection is connected to the bias-complexity tradeoff because the selected feature set determines part of the model's complexity. The two-feature design works in a smaller feature space, while the five-feature design works in a larger feature space. Moving from two features to five features therefore changes the complexity side of the tradeoff. The design question is not whether the larger model is automatically better; it is what is gained and what becomes more demanding when the feature space grows.
Practical Design Tradeoffs
| Design choice | Practical advantage | Practical disadvantage |
|---|---|---|
| Fewer features | The model works in a smaller feature space. | The model does not use the other listed features, such as age, physical activity, and income. |
| More features | The model uses all five listed patient features rather than only two. | The model works in a larger feature space, making the complexity side of the design more demanding. |
Mistakes in Model Comparison
Treating the number of features as unrelated to dimensionality.
The number of selected features determines the dimensionality. The first model uses a two-dimensional space, while the second uses a five-dimensional space.
Fix:
Count the selected features before comparing the models.Assuming that using more features automatically makes the model better.
The comparison is about the pros and cons of different complexity levels, not an automatic winner.
Fix:
Explain both the larger feature space and the demands associated with greater complexity.Discussing the prediction task without discussing feature selection.
The central design question includes both the algorithm choice and the feature set.
Fix:
Name the selected features and connect them to the resulting feature-space dimensionality.
Check Your Design Reasoning
A heart-attack prediction algorithm uses blood pressure, body-mass index, and age. What is the dimensionality of its feature space? Then compare it with the two-feature design and explain one way the complexity discussion changes.
Hints
- Count the selected features.
- The two-feature design uses blood pressure and body-mass index.
- Use the language of smaller or larger feature spaces and the bias-complexity tradeoff.
A complete answer should identify the three-dimensional feature space, compare it with the two-dimensional space, and explain that changing the feature count changes the complexity side of the bias-complexity tradeoff.
Feature Selection Summary
- Selecting features determines the dimensionality of the feature space.
- Using blood pressure and body-mass index creates a two-dimensional feature space.
- Using blood pressure, body-mass index, age, physical activity, and income creates a five-dimensional feature space.
- Fewer features create a smaller feature space, while more features create a larger feature space and a more demanding complexity comparison.
- Feature selection is part of the bias-complexity tradeoff, so models should be compared by explaining their respective advantages and disadvantages rather than naming an automatic winner.
Key Takeaways
- The selected feature set determines the dimensionality of a learning algorithm's feature space.
- The blood-pressure-and-BMI design uses two dimensions; the design using all five listed patient features uses five dimensions.
- Fewer features produce a smaller feature space but leave out some available measurements.
- More features include more of the listed measurements but create a larger and more demanding feature space.
- Feature selection should be analyzed as part of the bias-complexity tradeoff, not as a decision with an automatic winner.