Hyperparameter Tuning with Cross-Validation

Imagine you have already tried a few settings for a model, and now you want to tune its hyperparameters with a grid search. You will use cross-validation to compare the candidate settings. Before any of that starts, you prepare the data: you scale the features so that they sit on comparable ranges. Doing this step once, at the very beginning, feels efficient and harmless.

A common assumption says this is fine. The reasoning goes like this: scaling is just a bit of arithmetic applied to the features, so the order in which it happens should not matter. The source material identifies this exact belief as a misconception. This article traces what happens to the data step by step, so you can see where the belief breaks down and why the order of preprocessing and splitting changes how much you can trust the numbers that come back.

Follow the data through the wrong order first. Start with the full training set. Before cross-validation begins, you learn a scaling from the entire training set and apply it to every row. Only after that do you divide the already-transformed data into splits. Each split takes a portion of the training set and designates it as the test part of that split.

Here is the subtle detail. Even though a portion is held out during fitting, that portion still belongs to the training set as a whole. When the scaling was learned from the entire training set, it absorbed information from every row, including the rows that later serve as test parts. By the time cross-validation begins, the held-out data has already shaped the transformation applied to the training data. The test part is no longer independent of what the model learns from.

The purpose of the test part is to simulate what new data will look like to a model trained on the training part. That simulation only works if the test part behaves like unseen data. Once preprocessing has already drawn on the test part, it no longer behaves that way. The splits stop being an honest stand-in for future data, and the model appears to perform better than it would on data it has never encountered.

The remedy is a change in ordering. The dataset should be split during cross-validation before any preprocessing takes place. Any process that extracts knowledge from the dataset, such as learning a scaling, should only ever be applied to the training portion of each split. The test portion is then transformed using what was learned from the training portion, not using statistics that include it.

In other words, cross-validation becomes the outermost loop in the processing pipeline, and everything that learns from data sits inside it. Read the diagram below from top to bottom. The split happens first, the preprocessing is fit only on the training part, both parts are transformed, the model is trained, and only then is the test part scored.

Cross-validation is a loop, and the correct workflow depends on that loop running the whole process again for every split. Each fold has its own train and test partition. Because the test part changes from fold to fold, what the model sees as unseen also changes. The preprocessing that is learned must change with it, which is why it belongs inside the loop rather than outside.

Read the loop as a repeated cycle. For every split, the scaling is learned only from that split's training part, the training part and the test part are each transformed, the model is trained, and only then is the test part scored. Because no test information enters the preprocessing step, the score reflects how the model handles data it has not been allowed to influence.

The first consequence of leakage is that the scores come out overly optimistic. The model appears to perform better than it would on data it has never encountered, because the test part was no longer a faithful stand-in for future data. A reported score that looks excellent may simply reflect the leak rather than real generalization.

The second consequence is less obvious and can be more damaging. Grid search chooses settings based on the scores it sees. If those scores are inflated, the search can pick a setting that benefited from the leak rather than the one that truly generalizes best. Leakage therefore does not only misreport the final number; it can also lead to suboptimal parameter selection.