Error Decomposition

The Design Question

When designing a learning algorithm, choosing the algorithm is only part of the decision. You must also decide which available features the algorithm should use. In the heart-attack prediction example, the available features are blood pressure, body-mass index, age, level of physical activity, and income. Selecting some or all of these features changes the dimensionality of the feature space.

Counting Feature Dimensions

A feature space has one dimension for each selected feature. If an algorithm uses blood pressure and body-mass index, it uses two dimensions: one for blood pressure and one for body-mass index. If it uses blood pressure, body-mass index, age, physical activity, and income, it uses five dimensions. The dimensionality therefore comes directly from the number of selected features, not simply from the total number of features that happen to be available.

Two Algorithms, Two Representations

Consider two possible algorithms for predicting whether a patient will suffer a heart attack. Algorithm A uses only blood pressure and body-mass index. Algorithm B uses blood pressure, body-mass index, age, physical activity, and income. Algorithm A represents each patient using two selected features, so it works in a two-dimensional feature space. Algorithm B represents each patient using five selected features, so it works in a five-dimensional feature space.

Bias and Complexity

The difference between the two algorithms is the complexity side of the bias-complexity tradeoff. The two-feature algorithm works in a smaller feature space. The five-feature algorithm works in a larger feature space. Moving from the smaller space to the larger one changes what the algorithm must handle and makes the design more demanding. Moving in the other direction creates a simpler representation, but it also means using fewer of the available features. The correct analysis is therefore a comparison of tradeoffs, not an automatic claim that one choice is always better.

Practical Feature Tradeoffs

Using fewer features gives the algorithm a smaller feature-space representation. That is a practical advantage when a simpler design is useful. The corresponding disadvantage is that the algorithm is built from a narrower feature selection. Using more features gives the algorithm a larger representation and can make the design more demanding. Its practical advantage is that the algorithm uses more of the available feature set; its disadvantage is the greater complexity associated with the larger feature space. These are tradeoffs to analyze, not evidence that either the two-feature or five-feature choice is universally superior.

Check Your Reasoning

Frequent Reasoning Errors

• Confusing available features with selected features.

  Dimensionality is determined by the features the algorithm uses, not by every feature that is available.

  Fix: Count the selected features when identifying the feature-space dimensionality.

• Calling the two-feature algorithm five-dimensional because five patient features were listed.

  The two-feature choice creates a two-dimensional feature space.

  Fix: Match the dimensionality to the selected feature set: two selected features means 2D.

• Assuming that more features automatically make an algorithm better.

  The comparison requires analyzing the pros and cons of different complexity levels rather than declaring an unconditional winner.

  Fix: State that the five-feature choice creates a larger feature space and evaluate that choice as part of the bias-complexity tradeoff.

• Discussing complexity without naming the feature-space change.

  The central change is the move between a smaller two-dimensional space and a larger five-dimensional space.

  Fix: Explain the selected features first, then connect their count to dimensionality and complexity.

Key Takeaways

1. The number of selected features determines the dimensionality of the feature space.
2. Using blood pressure and body-mass index creates a two-dimensional feature space.
3. Using blood pressure, body-mass index, age, physical activity, and income creates a five-dimensional feature space.
4. Fewer and more features each involve practical advantages and disadvantages connected to smaller or larger feature-space complexity.
5. Feature selection is part of the bias-complexity tradeoff, so the right comparison explains the tradeoffs instead of assuming that one feature count is always best.