Model-Based Feature Selection Strategies

This article begins with code rather than a definition. A model-based feature selector reduces a training set by deciding which feature columns to keep. The pattern you will read below comes from a worked exercise in which a feature selector reduces a training set from 80 features to 40. Read the lines once through, then work out what each line should do before you read the explanation that follows.

The source snippet does not show how select was created, so the selector is assumed to have been set up earlier in the same exercise. What the snippet does show is the complete pattern: one call to fit, one call to transform, and two print statements. The shape printout is the evidence you will use to check whether the selection behaved as expected. The code above is a reconstruction of that pattern, written to match the calls the source describes.

The method select.fit(X_train, y_train) is the step where the selector examines the training data. It receives both the feature matrix X_train and the target values y_train, because a model-based selector decides which features matter by using a model that is trained on those inputs and targets. The fit call changes the state of the select object, but it does not return a new dataset.

The method select.transform(X_train) is the step that applies the decision. It takes the feature matrix and returns a new array that contains only the features the selector kept. In the source, that result is stored in a new variable, X_train_l1, so the original X_train is left unchanged. This separation matters when you debug: if the shape of X_train_l1 is wrong, the problem lies in the fit step, the transform step, or the data that was passed to them.

The order of the calls is fixed. The selector must be fitted before transform is called, because transform relies on the decision made during fit. The training matrix enters the process in two places: it is used to learn the selection during fit, and it is then passed to transform to produce the reduced array. The diagram below traces that flow.

Shape is the most useful first check in this exercise because it reports two facts at once: how many samples remain and how many features remain. A shape is written as a pair of numbers, rows first and then columns. Comparing the two shapes printed by the script tells you exactly what the transform step did.

After the reduced array has been checked, the source looks at which features were selected. The call select.get_support() returns a mask, an array that records, for each original feature, whether the selector kept it. The mask has one entry per original feature, so each position answers the question: was this feature kept?

The mask is reshaped with mask.reshape(1, -1) before it is displayed. This turns the one-dimensional mask into a single row with one column per original feature. The matshow function then draws that row as a strip of cells, and the cmap='gray_r' argument chooses the reversed grayscale palette in which True cells are black and False cells are white. The source's tick marks run from 0 to 70 along the horizontal axis, which marks the range of feature positions across the strip.

Because the mask is drawn as a single row, the horizontal position of each cell corresponds to an original feature position, not to a sample. Position 0 is the first original feature, the next position is the second, and so on. Reading the strip from left to right is therefore a scan across the feature columns of the original matrix. A black cell means that feature was kept by the selector, and a white cell means it was dropped.

The source labels this horizontal axis "Sample index". That label describes the strip in a misleading way, since the cells represent features and not samples. When you write your own version of this plot, change the label so that it matches what the axis actually shows, such as "Feature index".

When a selection script misbehaves, locate the failure by comparing each step's output with its expected output, starting from the fit call and moving forward. Check the output of each step in order, and stop at the first step whose result differs from what you expected. The problem originates there or in the data that was passed into it.