Selecting the Most Relevant Features

This article follows a short feature selection exercise from start to finish. The exercise begins with a training set that has 80 features, and the selector reduces it to 40 features. The number of samples does not change. Before you read the explanation, try to work out what the shape of the reduced training matrix should be, and which part of the printed output would tell you that the selection worked.

Model-based feature selection uses a selector that works in two distinct stages. First it is fitted with fit(X_train, y_train). Then it is applied with transform(X_train) to produce a reduced feature matrix. The two stages are separate, and the order between them matters.

The fit call is where the selector examines the training data. It receives both the feature matrix X_train and the target values y_train, because a model-based selector decides which features matter by using a model that is trained on those inputs and targets. The fit call changes the internal state of the selector object, but it does not return a new dataset. Nothing reduced has been produced yet at this point.

The transform call is where the decision made during fit is applied. It takes the feature matrix and returns a new array that contains only the features the selector kept. In the exercise, that result is stored in a new variable called X_train_l1, so the original X_train is left unchanged. Keeping the original and the reduced arrays in separate variables is useful when you debug, because you can compare them directly.

Shape is the most useful first check in this exercise, because it reports two facts at once: how many samples remain and how many features remain. The script prints the shape of X_train and then the shape of X_train_l1. Comparing the two printed pairs tells you exactly what the transform step did.

After the reduced array has been checked, the exercise looks at which features were selected. The call select.get_support() returns a mask. The mask has one entry per original feature, and each entry records whether the selector kept that feature. A True entry means the feature was kept, and a False entry means it was dropped. Each cell therefore answers one question: was this feature kept?

The mask is then reshaped with mask.reshape(1, -1) before it is displayed. This turns the one-dimensional mask into a single row with one column per original feature. The matshow function draws that row as a strip of cells. The argument cmap='gray_r' selects a reversed grayscale palette in which True cells are black and False cells are white. The tick marks in the exercise run from 0 to 70 along the horizontal axis, which covers the positions of the features across the strip.

When a selection script misbehaves, do not start by guessing at the cause. Compare each step's output with its expected output, beginning at the fit call and moving forward. The first step whose output differs from what you expected is where the problem starts. Later steps can only show symptoms of that earlier fault.