Feature Selection

From Features to a Vector

A learning algorithm does not work directly with an abstract description of a problem. It works with a feature vector: a representation built from the original features. Feature manipulation changes that representation before it is given to the learning algorithm. In this article, feature selection is treated through the source concept of feature manipulation: transforming each original feature to create a resulting feature vector.

Tracing One Transformation

The example does not claim that one transformation is always correct. It shows the mechanism: original features are changed into a representation that the learning algorithm can use. The quality of that representation depends on the purpose of the manipulation, the selected learning algorithm, and the assumptions made about the problem.

Why Manipulate Features

Feature manipulation can serve at least two purposes. One purpose is to reduce approximation or estimation errors. Another is to obtain a faster algorithm. These are possible goals, not a guarantee that every transformation achieves both. A transformation should therefore be judged in relation to the learning problem rather than treated as universally beneficial.

Algorithm and Assumptions

A feature transformation is not universally good or bad. It is part of a larger design decision. First, the transformation produces the representation used by the learning algorithm. Second, the learning algorithm interprets that representation according to its own method. Third, prior assumptions about the problem influence whether the representation is appropriate. Changing the algorithm or changing those assumptions can therefore change which transformation is considered correct.

Normalization in Linear Regression

Normalization is introduced as a type of feature manipulation. Its motivation is presented through a linear regression problem using squared loss. In that setting, the data is described by a matrix whose rows are instance vectors and by a vector containing target values. Normalization changes the feature representation before that linear regression procedure uses it.

The before-and-after view is about the representation supplied to the model. It is not a claim that normalization has one universal numerical procedure or one universal effect. The source uses the linear regression and squared-loss setting to motivate normalization, so the learning algorithm and loss belong in the explanation.

Ridge Regression Data Flow

The normalization discussion also introduces ridge regression. In the stated linear regression setting, the rows of the matrix X are instance vectors, and y is a vector of target values. After feature normalization, the resulting instance vectors and their target values form the inputs to ridge regression. Ridge regression returns a vector based on those instance vectors and target values.

Mistakes to Avoid

• Treating feature manipulation as something the learning algorithm does directly to an abstract problem description.

  The algorithm works with a feature vector, and feature manipulation changes that representation before the algorithm receives it.

  Fix: Trace the order explicitly: original features, transformation, resulting feature vector, then learning algorithm.

• Assuming one feature transformation is best for every problem.

  The correct transformation depends on the learning algorithm and the prior assumptions about the problem.

  Fix: Evaluate the transformation together with the algorithm and the assumptions it is meant to support.

• Claiming that feature manipulation always reduces error.

  Reducing approximation or estimation errors is one possible purpose, not a universal guarantee.

  Fix: Describe error reduction as a goal that must be considered in context.

• Leaving target values out of the ridge regression setup.

  The stated setup includes both the rows of X, which are instance vectors, and y, which contains target values.

  Fix: Keep the normalized instance vectors and target values connected when describing the regression input.

Practice and Summary

1. Feature manipulation transforms original features into a resulting feature vector used by a learning algorithm. Its purposes may include reducing approximation or estimation errors and obtaining a faster algorithm. The appropriate transformation depends on the learning algorithm and prior assumptions about the problem. Normalization is presented as a feature manipulation motivated by linear regression with squared loss. In the associated ridge regression setup, the rows of X are instance vectors, y contains target values, and ridge regression returns a vector based on those inputs.