Preprocessing with Scalers

A common way to set up a model search goes like this: scale the whole dataset first, then split it, then run a grid search over candidate settings. The reasoning seems tidy. Every row receives the same transformation, so the model sees consistent inputs in every part of the search. It is easy to assume that a preprocessing step applied before the search is just bookkeeping and cannot change how the evaluation turns out.

This page follows one scaler through a workflow and watches where the test rows first influence it. The goal is to see how a leak enters, how it reaches the final score, and how a pipeline prevents it.

When you debug a leak, the first question is where the code first touches the full dataset. In the flawed workflow, the scaler is fit once, before any splitting happens. The scaler reads every row and stores the parameters it learns. Every later step inherits those parameters, including the grid search, which means every candidate setting is evaluated on inputs that were scaled using information from the test rows.

The diagram shows the key point: the leak is not in the grid search itself. It enters at the first point where the scaler is fit on the full dataset, which happens before any split. Once that happens, the grid search faithfully reports results computed from inputs that already contain test information.

The held-out fold is supposed to play the role of data the model has never encountered. A preprocessing step that learns from that fold, such as estimating the scale of the data, quietly gives the search a head start. The model never trains on the held-out rows directly, yet the evaluation still carries their influence, because the inputs were transformed using parameters that were computed with those rows included. The grid search then picks settings that look better than they really are.

The corrected workflow does not need a separate scaling step at all. You place the scaler and the model in one pipeline, which is a sequence of data processing steps, and you hand that pipeline to the grid search. Cross-validation then handles the scaler as part of the model. For each split, the scaler is refit using only the training portion of that split. The held-out portion is transformed with the parameters learned from the training portion, but it never contributes to learning them.

Read the sequence from top to bottom. The scaler's parameters are learned in step two, from the training portion alone. Step three only applies those parameters to the held-out portion. Because the held-out rows never reach the fitting step, the score in step five reflects a model that was prepared without seeing the test rows. This is the contrast with the earlier workflow, where the scaler saw all the data before the search began.

Cross-validation divides the rows into folds and rotates which fold is held out. The example below uses three folds for illustration. In each iteration, one fold plays the held-out role and the other folds form the training portion. The pipeline must refit the scaler for each of these assignments, using only the rows that are training in that iteration.

In split 1, Fold A is the held-out portion and Folds B and C form the training portion. The scaler must learn its parameters from Folds B and C only. In split 2, Fold B is held out, so the scaler is refit on Folds A and C. In split 3, the scaler is refit on Folds A and B. Each refit uses a different training portion, and each held-out fold is transformed only after the refit for its own split. No single scale is shared across the three iterations in a way that lets a held-out fold shape its own inputs.

Not every leak is equally damaging. The impact depends on the nature of the preprocessing step. Estimating the scale of the data from the test fold usually does not have a terrible impact. Using the test fold in feature extraction and feature selection, however, can lead to substantial differences in outcomes. A useful way to compare steps is to ask whether the step learns parameters from the data at all, and if so, how much of the test fold it absorbs.

The table makes one point clear. A scaler and a feature selection step both learn from the data, so both can leak if they are fit on the full dataset. The size of the damage differs, and the source treats feature extraction and feature selection as the cases where a leak is most likely to produce substantial differences. Whatever the step, the safe pattern is the same: fit it inside the pipeline, on training portions only.

The leak returns easily, because the flawed order of operations looks reasonable. The following mistakes are the ones most likely to reintroduce it.

The flawed and corrected workflows differ in one place: where the scaler is fit. Fitting it on the full dataset before splitting lets test rows shape the parameters used for every candidate. Placing the scaler inside a pipeline gives each split its own refit, using only that split's training portion, while the held-out portion is only transformed.