Univariate Feature Selection

A supervised dataset often contains more columns than a useful model needs. Some features carry real signal about the target, while others add noise, duplicate information, or simply make the model harder to train. A common way to deal with this is to let a supervised machine learning model do the sorting. Instead of judging each feature on its own, you fit a model and ask it how much each feature contributes. The features can then be ranked by that measure, and only the strongest ones kept.

As you read, keep one question in mind: what does a model need to hand back so that we can rank its features? The answer shapes the rest of this article. We will first look at the general idea, then at the two families of models the source highlights, and finally at the transformer that performs the selection in code.

The phrase keeps only the most important ones describes the final goal, but the mechanism has two distinct parts. First, a model produces a measure of importance for every feature. Second, those measures are used to rank the features and decide which to retain. The source states the requirement for the first part directly: the selection model needs to provide some measure of importance for each feature, so that the features can be ranked by that measure. If a candidate model cannot produce such a measure, it cannot drive selection in this way.

Follow one feature matrix from start to finish. The selection model does its work on the full feature set, producing one importance measure per feature. The ranking step then turns those measures into an ordering. Only after that does the selection decision narrow the set. The final modeling step receives the reduced set, which may or may not be the same kind of model that performed the ranking.

Decision trees and decision tree-based models provide a feature_importances_ attribute. The source describes this attribute as one that directly encodes the importance of each feature. That phrase matters: for these models, you do not need to derive an importance score from some other quantity. After the model is fitted, the importance values are available as an attribute, ready to be ranked.

The attribute is accessed on the fitted model object, not on the raw data. Each position in the importance array corresponds to a column of X, so index 0 refers to the first feature, index 1 to the second, and so on. Ranking the features means sorting these positions by their values.

The code shows the attribute being read rather than any particular set of numbers. Run it on your own dataset to see the values for that data.

Linear models take a different route. They do not expose a feature_importances_ attribute. Instead, they have coefficients, and these can also be used to capture feature importances by considering their absolute values. A coefficient is a signed number, positive or negative. Its sign tells you the direction of a feature's effect, but for ranking importance you care about how large the effect is, so the sign is set aside by taking the absolute value.

Here $w_{j}$ is the coefficient for feature $j$, and the importance score used for ranking is $|w_{j}|$. Features with larger absolute coefficients are ranked as more important under this measure.

The source also connects linear models to sparsity. Linear models with an L1 penalty learn sparse coefficients, meaning that only a small subset of features receives nonzero weight. That sparsity can be viewed as a form of feature selection built into the model itself. It can also serve as a preprocessing step: the features with nonzero coefficients are kept and passed on to another model.

Computing importance scores is only half the job. To use model-based feature selection in a workflow, the SelectFromModel transformer does the rest. In the scikit-learn style, a transformer is an object that learns something from training data and then transforms data. Here the thing it learns is which features pass an importance test, and the transformation removes the features that fail.

The first printed shape is the original matrix, and the second shows the reduced one. The number of rows does not change, because the transformer removes features, not observations. The decision step depends on a threshold, and the exact threshold options are part of the transformer's configuration. Read the documentation for the version you have installed, and run the code to see how many columns were kept for your dataset.

The source contrasts model-based selection with univariate selection. Univariate methods judge each feature separately, one at a time. Model-based selection instead considers all features at once, because the model is fitted on the full feature set. This is the reason the source says model-based selection can capture interactions, provided the model is able to capture them.