Model-Based Feature Selection

Many real datasets arrive with far more input columns than a model can comfortably use. Some of those columns carry useful information about what you want to predict. Others are noise: values that vary randomly and have no real connection to the target. Feature selection is the family of techniques for deciding which columns to keep before a model is trained on them.

Two decisions are needed. First, you choose a statistical test. The source names f_classif as the default for classification and f_regression for regression. Second, you choose a rule for discarding features based on the p-values that test produces. Every discarding method uses a threshold, and the methods differ in how they compute that threshold.

Because each test is independent, two features can end up with very different results. One column may show a clear connection to the target while another, placed right beside it, shows none. The selector only sees each result separately and then applies the threshold to each one.

The source names two of the simplest discarding methods. SelectKBest selects a fixed number k of features. SelectPercentile selects a fixed percentage of features. The difference lies only in how the size of the kept set is specified. With SelectKBest you say how many columns survive, regardless of how many columns the dataset has. With SelectPercentile you say what share of the columns survives, so the count adjusts automatically as the dataset grows or shrinks.

The percentile argument is a number from 0 to 100 that sets the share of features to keep. The source uses 50, meaning half of the features. Choosing between a fixed count and a fixed percentage is a judgment call. A percentage adapts to the dataset's width, which helps when the same pipeline must handle inputs of different sizes. A fixed count gives direct control over the model's input size, which helps when a downstream constraint sets a maximum number of columns.

The usage pattern has three steps. You create the selector with a percentile, fit it so that it can score the features against the training targets, and then transform the training data to keep only the selected columns. The code below follows that pattern. The source snippet does not show how the selector was created, so the setup line is assumed here as an illustration.

The method select.fit(X_train, y_train) is the step where the selector examines the training data. It receives the feature matrix X_train and the target values y_train. The fit call changes the internal state of the select object, but it does not return a new dataset. The method select.transform(X_train) then applies the decision. It returns a new array containing only the kept features, which the source stores in a new variable called X_train_l1.

The order matters. The selector must be fitted before transform is called, because transform relies on the decision made during fit. Calling transform on an unfitted selector has no decision to apply, so the fit step always comes first.

Shape is the most useful first check in this exercise because it reports two facts at once: how many samples remain and how many features remain. A shape is written as a pair of numbers, rows first and then columns. Comparing the shapes printed before and after transform tells you what the transform step did.

In the source exercise, the feature count drops from 80 to 40 while the 284 samples remain unchanged. The shape printout is the main evidence that the selection worked. If the row count changed, something other than feature selection has gone wrong, because this step is meant to remove columns only.

Transforming the data tells you how many features survived, but not which ones. The get_support method answers the second question. It returns a Boolean mask, an array with one True or False value per original feature. True marks a feature that was kept and False marks a feature that was removed. The position of each entry in the mask matches the position of that feature in the original input, so the mask is a direct map from original column positions to keep-or-drop decisions.

Reading the mask this way lets you check the selector's decisions against the original column order. If you know that a column at position 2 is important, you can look up its entry directly instead of guessing from the transformed array, which has lost the original positions.

After the reduced array has been checked, the source looks at which features were selected. The mask is reshaped with mask.reshape(1, -1) before it is displayed. This turns the one-dimensional mask into a single row with one column per original feature. The matshow function then draws that row as a strip of cells. The cmap='gray_r' argument chooses a reversed grayscale palette in which True cells are black and False cells are white.

Because the mask is drawn as one row, the horizontal positions in the plot correspond to original feature positions. The source's tick marks run from 0 to 70 along the horizontal axis, which covers the range of feature positions across the strip. Read the strip from left to right: each black cell is a kept feature and each white cell is a removed feature.

Removing features only matters if the model does as well or better afterward. The source checks this by training logistic regression twice on the same split: once on all features and once on only the selected features, then scoring both on the test set. In the source's synthetic example, the reduced feature set scored slightly higher. Removing the noise columns improved performance, even though some genuine features were lost along the way.

The source's dataset was a breast cancer dataset with 30 real features, to which 50 columns of random noise were appended, giving 80 columns. The selection was imperfect: some real features were dropped and some noise columns were kept. The recovery was not perfect, yet the reduced set still scored higher.

Univariate selection is still useful in two situations the source names: when there are so many features that building a model on all of them is infeasible, and when you suspect that many features are completely uninformative.

When a selection script misbehaves, locate the failure by comparing each step's output with its expected output, starting from the fit call and moving forward. The shape printouts act as checkpoints. If X_train_l1 has the wrong number of columns, the problem lies in the fit step, the transform step, or the data passed to them. Work through the chain in order rather than guessing.