Feature Selection Methods

Cross-validation is supposed to tell you how well a model will perform on data it has never seen. That promise only holds if the test folds stay untouched by the training process. When some information from the test data quietly shapes how the model is built, the score you measure is inflated. This failure is called information leakage, and it is the central idea of this article.

Many people assume that a preprocessing step such as feature selection is harmless as long as the model itself is evaluated with cross-validation. The reasoning goes like this: the selector only chooses columns, the Ridge regressor is what does the learning, and so the cross-validation score reflects the regressor's real ability. This belief is what we are going to test.

The article works through a deliberately unusual case. The data contains no real signal at all, so any model that scores well on it is scoring well for the wrong reason. Watching how such a score can appear is the clearest way to understand how leakage works and where to look for it.

The starting point is a synthetic regression task with 100 samples and a large number of features. Every feature is drawn independently from a Gaussian distribution, and the response is also drawn from a Gaussian distribution. Because of how the data was created, there is no relationship between the features and the target. In principle, nothing can be learned from it.

The leak in this case is subtle because nothing looks wrong on the surface. The code still splits the data into folds, still trains a regressor on training rows, and still scores it on held-out rows. The problem is hidden earlier in the workflow, in a step that touched every row before the split happened.

The improper workflow runs in a fixed order. First, feature selection is fitted on the full dataset, so every row, including the rows that will later serve as test folds, contributes to deciding which features survive. Second, the reduced dataset is handed to cross-validation, which splits it into folds and fits the Ridge regressor on each training portion. The diagram below traces how data flows through those two stages and shows where the test rows first affect the selector.

The highlighted step is where the leak enters. The test folds were not held out before selection, so the selector could pick features that happen to be correlated with the target across both the training and the test rows. When the Ridge regressor is later scored on those test rows, the score reflects a pattern that was partly chosen using the test rows themselves. The regressor never saw the test rows during fitting, but the features it was given were already shaped by them.

The reason this matters is that random data always contains some accidental correlations. Among thousands of random features, some will be correlated with the target purely by chance. A selector that looks at all the rows will find those chance correlations and keep them. The test folds then appear to confirm them, because the test rows are part of the same pool that produced the selection.

Leakage is not caused by the regressor. In this example the divergence is in the earlier feature selection step, and the Ridge regressor behaves normally on the data it receives. This is why checking the regressor first would waste time: the error is upstream of it.

The difference between the two workflows is only the position of one step, but that step decides whether test rows influence training. In the improper version, selection happens before the split. In the proper version, the split happens first, and selection is repeated for each training fold. The flowchart below places the two sequences side by side so you can see which step moves earlier or later.

In the proper version, selection moves later, after the split and inside each training portion. The test fold is never consulted while choosing features. The same rule applies to any other step that fits itself to data: if it is fitted on rows that later appear as test rows, the score can no longer be trusted as an estimate of performance on unseen data.

A pipeline is a sequence of data processing steps that run together. When a pipeline with a feature selector followed by a Ridge regressor is passed to cross-validation, each fold repeats the whole sequence using only that fold's training rows. The selector is therefore fitted inside the loop, and the test fold is never consulted while choosing features. The diagram below shows how the same selector is fitted on training data and then only applied to the test data.