Feature Selection Fundamentals

Feature selection answers a practical question. When you are unsure which columns of your data to feed into a machine learning algorithm, can an automatic method pick the useful ones for you? Recursive Feature Elimination, usually called RFE, is one such method. It works by recursively eliminating features until a specified number is reached. In the source example that number is 40, so the method starts with the full set of columns and keeps removing them until 40 remain.

Automatic feature selection is most useful when you do not know in advance which inputs matter. Reducing the number of features also has practical benefits. A smaller feature set means less data to process at prediction time, and fewer inputs to explain when you need to justify what a model does. Keep these benefits in mind, because the rest of this article shows that they come at a real cost in computation.

RFE wraps an estimator, which is the model it trains again and again while deciding what to remove. In the source setup the estimator is a RandomForestClassifier with 100 trees and a fixed random_state of 42. On each round RFE fits the estimator on the features that are still in play, ranks those features, and drops the weakest. Then it repeats the same fit-rank-eliminate cycle on the smaller set.

The loop explains the cost that the source warns about. Because the random forest is trained once for each feature that is dropped, the total work grows with the number of eliminations. The more features RFE must remove before reaching its target, the more models it trains. The source describes RFE as computationally expensive for exactly this reason.

The setup has three parts: the estimator RFE will retrain, the stopping point given by n_features_to_select, and the fit call that runs the loop. The code below follows the logic of the source example, reordered so every name is defined before it is used. It assumes that X_train, X_test, y_train, and y_test already exist, and that the feature table has been split into training and test portions.

After fit has run, get_support returns a mask with one entry per original feature. The mask is positional. Its first entry describes the first original column, its second entry describes the second column, and so on. A True entry means that column survived elimination and a False entry means it was dropped. Reading the mask is therefore a matter of lining it up against the original column list.

Once RFE has fit, the selector can shrink both the training and test data down to the chosen features. The fit step is the only step that learns which features to keep, and it does so using training data alone. Calling transform on the training data and on the test data then applies that already-learned choice. The columns come back in the correct order, so the reduced arrays line up with each other.

The source compared RFE with univariate selection and with model-based selection. In its experiment RFE chose features that performed better than either alternative. It also missed one feature that mattered, so even its best selection is not guaranteed to be perfect. The performance gain came at a cost in time, because RFE trains many models where model-based selection trains one.

RFE is worth its cost when the quality of the selected features matters more than the time spent finding them, and when a single-pass method is not good enough for your data. When speed of selection is the priority, or when you are unsure whether the extra accuracy is needed, a cheaper approach is a reasonable first choice.