Univariate Feature Selection with f_classif and f_regression

Many real datasets come with far more input columns than a model can comfortably use. Some of those columns carry useful information about what you want to predict. Others are noise: values that vary randomly and have no real connection to the target. Feature selection is the family of techniques for deciding which columns to keep before a model is trained on them.

This article focuses on one simple approach available in scikit-learn. A univariate method scores each feature, SelectPercentile turns those scores into a keep-or-drop decision based on a fixed share of the columns, and you then check the result. Keep one question in mind as you read: if you were told to keep half of a set of columns, what would you use to decide which half is worth keeping?

The word univariate means one variable at a time. Every column is tested against the target separately, without asking how it interacts with the other columns. The result of each test is a p-value. A high p-value means the feature is unlikely to be related to the target, so it becomes a candidate for removal. A low p-value means the feature looks related to the target and is more likely to be kept.

The method needs two decisions. First, you choose a statistical test. The default for classification is f_classif, and f_regression is the matching choice for regression. Second, you choose a rule for discarding features based on the p-values that test produces. Every discarding rule uses a threshold, and the methods differ in how they compute that threshold.

Read the diagram as a single feature's journey. The feature is never compared with its neighbours at this stage. Its own p-value alone decides whether it is a candidate for removal, and the threshold set by the discarding rule determines where the line falls.

The two simplest discarding rules differ only in how the size of the kept set is specified. SelectKBest selects a fixed number k of features, so you say how many columns survive no matter how many columns the dataset has. SelectPercentile selects a fixed percentage of features, so you say what share of the columns survives and the count adjusts automatically as the dataset grows or shrinks.

The usage pattern has three steps. First, you create the selector with a percentile. The percentile argument is a number from 0 to 100 that sets the share of features to keep; 50 means half of the features. Second, you fit the selector on training data so that it can score the features against the training targets. Third, you transform the training data, which keeps only the selected columns.

Notice that fit uses the training targets, so the selector learns which features to keep from training data only. The same fitted selector is then used to transform the test data, so the model sees exactly the same columns it was trained on. Fitting on the training split alone is what keeps the test set honest.

Transforming the data tells you how many features survived, but not which ones. The get_support method answers the second question. It returns a Boolean mask: an array with one True or False value per original feature. True marks a feature that was kept, and False marks a feature that was removed.

The position of each entry in the mask corresponds to the position of that feature in the original input. The mask is therefore a direct map from original column positions to keep-or-drop decisions. Its length always equals the number of input columns, not the number that survived.

A larger illustration uses a breast cancer dataset with 30 real features. Fifty columns of random noise are appended, so the input has 80 columns in total. Running SelectPercentile with percentile 50 keeps about half of them. The mask is then plotted as a single horizontal strip of 80 cells, in order, with black for True and white for False.

Reading that strip across, most of the original 30 features appear black, meaning kept, while most of the 50 noise columns appear white, meaning removed. The recovery is not perfect: some real features were dropped and some noise columns were kept. The strip is a visual reading aid, so when you need exact positions, go back to the Boolean mask values themselves.

Removing features only matters if the model performs as well or better afterward, so the check has to be made explicitly. In the walkthrough, logistic regression is trained twice on the same train-test split: once on all features and once on only the selected features. Both models are then scored on the test set.

In that synthetic example the reduced feature set scored slightly higher. Removing the noise columns improved performance even though some genuine features were lost along the way. The walkthrough itself warns that this was a very simple synthetic setup and that results on real data are usually mixed.

Univariate selection remains useful in two situations. The first is when there are so many features that building a model on all of them is infeasible. The second is when you suspect that many features are completely uninformative. Outside those situations, the benefit is uncertain, and it can be small or absent.

Because each feature is judged alone, the method can miss features that matter only in combination with other features. Two columns that are weak individually may be strong together, but the univariate test never asks how they interact. This is a limit of the method's design, not a bug in any one setting.