Model-Based Selection

Suppose you have a table with many candidate columns and you want to train a model on only the most useful ones. Some columns carry real signal about the target, while others add noise, cost time, or make the model harder to interpret. Choosing which columns to keep is called feature selection. This article focuses on one family of approaches, model-based selection, and on a particular kind of it that repeats the process many times. The central idea is that the decision about which features matter is handed to a model, and the model's own view of feature importance guides the choice.

Earlier methods for choosing features took one of two shapes. Univariate testing looks at each feature on its own, without using any model. Model-based selection trains one model and uses it to decide which features to keep. In both cases, the selection decision comes from a single pass over the data. Iterative feature selection asks a different question. Instead of making one decision, it builds a whole series of models, and each model in the series uses a different number of features. The selection process moves through that series, using what each model reveals to guide the next step.

The practical difference is in what the final choice is based on. A univariate test judges each column in isolation, so it cannot see how two columns work together inside a model. A single model-based selection uses one model's view of importance, but that view is taken once, from one set of inputs. An iterative method revisits the question several times, each time with a smaller or larger set of features, so the final subset reflects what the model reveals as the feature set changes.

Iterative feature selection has two basic methods. They start from opposite ends of the feature set and move toward a smaller or larger subset, depending on the method. Both continue building models until a stopping criterion is reached. The choice between them determines where the search begins. The diagram below shows how the feature set changes at each step in each direction.

RFE is easiest to understand by following a single run. The loop below shows one round: the current features are used to train a model, the model's importance scores are used to choose a feature to discard, and the reduced set becomes the input to the next round. The output of each model is not the end of the process. Its importance scores feed a decision that changes the feature set, and the changed set feeds the next model.

The example also shows why RFE rebuilds the model at every round instead of ranking everything once. Each round built a fresh model on a smaller set, so importance scores can change as features are removed. A feature that looked weak with five companions may look different once others are gone.

RFE can discard the least important feature only if the model can say which features are important. The source states that RFE depends on the selecting model being able to determine feature importance, which is the same requirement seen in model-based selection. A model that cannot rank its inputs gives RFE nothing to act on. The choice of model is therefore part of designing the method. The source's example uses a random forest model, the same model used earlier in the discussion of model-based selection. The selection is only as informed as the importance measure the model provides.

Because iterative methods build a series of models, they are much more computationally expensive than the methods discussed earlier. A univariate test may examine each feature once. A single model-based selection trains one model. An iterative method trains many models, and each one must be fit before the search can decide what to do next. The total work grows with the number of models in the series, not with a single training run.

In the five-feature example above, RFE trained three models: one on five features, one on four, and one on three. A method that removes one feature per round and stops at two remaining features needs a model fit for each size between the starting count and the target. As the feature count and the number of rounds grow, the number of fits grows with them, and each fit must finish before the next decision can be made.