Cross-Validation and Model Evaluation

Cross-validation is meant to answer a specific question: how well will a model perform on data it has never seen? It does this by repeatedly holding out part of the data as a test fold, training on the remaining rows, and scoring on the held-out rows. The answer it gives is only trustworthy if the test folds stay untouched by the training process. When that separation breaks, the score you measure is inflated, and it can look like a success even when the model has learned nothing useful.

This article works through a deliberately unusual case. The data contains no real signal at all, so any model that scores well on it is scoring well for the wrong reason. Watching how such a score can appear is the clearest way to understand how leakage works and where to look for it.

Many people assume that a preprocessing step such as feature selection is harmless as long as the model itself is evaluated with cross-validation. The reasoning usually goes like this: the selector only chooses columns, the Ridge regressor is the part that actually learns, and so the cross-validation score reflects the regressor's real ability. This is the belief the case study is designed to test.

The starting data is a synthetic regression task with 100 samples and a large number of features. Every feature is drawn independently from a Gaussian distribution, and the response is also drawn from a Gaussian distribution. Because of how the data was created, there is no relationship between the features and the target. In principle, nothing can be learned from it, so an honest evaluation should show performance close to nothing.

The leak in this case is subtle because nothing looks wrong on the surface. The code still splits the data into folds, still trains a regressor on training rows, and still scores it on held-out rows. The problem is hidden earlier in the workflow, in a step that touched every row before the split happened. A reader who only checks that the splitting, training, and scoring look correct will miss it.

The improper workflow runs in a fixed order. First, feature selection is fitted on the full dataset, so every row, including the rows that will later serve as test folds, contributes to deciding which features survive. Second, the reduced dataset is handed to cross-validation, which splits it into folds and fits the Ridge regressor on each training portion. The diagram below traces how data moves through those two stages.

The highlighted selection step is where the leak enters. The test folds were not held out before selection, so the selector could pick features that happen to be correlated with the target across both the training and test rows. When the Ridge regressor is later scored on those test rows, the score reflects a pattern that was partly chosen using the test rows themselves.

Random data makes the leak easy to see. Among thousands of random features, some will be correlated with the target purely by chance. Selection done on all the rows will find those chance correlations and keep them, and the test folds will then appear to confirm them. The confirmation is not evidence of learning, because the pattern was manufactured by the selection step rather than discovered in unseen data.

This is why the source's observation is useful beyond this one case. A score that looks too good, on data that should contain no signal, is a direct signal that some step has been fitted on rows it should not have seen. The strength of the score does not tell you what went wrong; only tracing the workflow does.

A pipeline is a sequence of data processing steps that run together. In this case, the pipeline contains a feature selector followed by a Ridge regressor. When that pipeline is passed to cross-validation, each fold repeats the whole sequence using only that fold's training rows. The selector is therefore fitted inside the loop, and the test fold is never consulted while choosing features.

The key change is not the choice of model or the scoring rule. It is the position of the selector. By moving selection inside the loop, every fold makes its own feature choices from its own training rows, and the held-out rows enter the process only when the finished pipeline is scored on them.

In the source's comparison, the pipeline version produces a far less flattering number, and that is the expected outcome, not a failure. Once the selector can only see training folds, the chance correlations it finds in those folds do not carry over to the held-out data. A realistic estimate on random data should be close to nothing, and the pipeline gives exactly that kind of estimate. The lower score is the more honest one.

When a cross-validation score looks too good, the useful question is not whether the model is good but where the result started to diverge from what the data can support. Trace the run step by step and mark every point where a transformation or selection is fitted. For each one, ask which rows it saw. If any step was fitted on rows that later appear as test rows, you have located the leak.