Recursive Feature Elimination in Practice

When a dataset has many features, a common task is to keep the useful ones and drop the rest. Earlier approaches make this choice in a single pass over the data: the decision about each feature comes from one calculation or one trained model, and then the process is finished. Iterative feature selection takes a different route. Instead of making one decision, it builds a whole series of models, each using a different number of features, and lets what each model reveals guide the next step.

Univariate testing looks at each feature on its own, without using any model. Each feature is judged separately, so the feature's relationship to the target is measured in isolation from the other features. Model-based selection takes a step toward considering the features together: it trains one model and uses that model to decide which features to keep. In both cases, the selection decision comes from a single pass over the data.

Iterative feature selection asks a different question. Rather than deciding once, it builds a series of models, and each model in the series works with a different number of features. The selection process moves through that series, using what each model shows to choose the next step. The diagram below places the three approaches side by side so the difference in how many models each one trains is easy to see.

Iterative feature selection has two basic methods. They start from opposite ends of the feature set and move toward a smaller or larger subset, depending on the method. One method starts with no features and adds them one by one. The other starts with all features and removes them one by one. Both keep building models until a stopping criterion is reached, and the choice between them determines where the search begins.

Recursive feature elimination (RFE) is a removal-based iterative method. It begins with all features and builds a model using them. That model is then used to find the least important feature, which is discarded. A new model is built using all of the remaining features except the one just discarded. The cycle repeats: build a model, discard the least important feature, build again. RFE stops once only a prespecified number of features are left.

The important point is that the output of each model is not the end of the process. Its importance scores become the input to a decision that changes the feature set, and the changed feature set becomes the input to the next model. Data flows around this loop until the stopping condition is met. The flowchart below shows that loop.

The example below is an illustrative one written for this article. The features are named A, B, C, and D, and the importance scores are invented to show the mechanics. The goal is to reduce four features to two. Follow what remains after each round, and notice that the scores are recalculated from scratch each time a model is rebuilt.

RFE can discard only the least important feature if the model can say which features are important. The requirement is that the model used for selection has some way to determine feature importance, the same requirement that applies to model-based selection. A model that cannot rank its inputs gives RFE nothing to act on, because the central step of the loop, discarding the weakest feature, has no basis for choosing.

The source's example uses a random forest model, the same model used earlier in the discussion of model-based selection. Choosing the model is therefore part of designing the method. The selection is only as informed as the importance measure the model provides, so the quality of the ranking directly controls the quality of every removal decision.

Because a series of models is built, iterative methods are much more computationally expensive than the methods discussed previously. A univariate test may examine each feature once. A single model-based selection trains one model. An iterative method trains many models, and each one must be fit before the search can decide what to do next. The total work grows with the number of models in the series, not with a single training run.