Preprocessing Within Machine Learning Workflows

Preprocessing that learns from data, such as scaling, must only see the training portion of each cross-validation fold.

Programming

Interactive lab

Try it: SELECT, WHERE, ORDER BY, LIMIT

How a SELECT query tests every row against the WHERE condition (keeping only rows where it is TRUE, not FALSE or NULL), then picks columns, removes duplicates with DISTINCT, sorts with ORDER BY and cuts with LIMIT.

How it works

FROM: start with every row of the table.
WHERE: evaluate the condition on each row. Comparisons with NULL give NULL (unknown); NOT, AND and OR follow three-valued logic, with AND binding tighter than OR. Only rows where the whole condition is TRUE are kept.
SELECT picks the listed columns; DISTINCT then drops result rows that repeat an earlier one.
ORDER BY sorts by each key in turn, ASC or DESC; SQLite puts NULL first when ascending and last when descending.
LIMIT n keeps only the first n rows of the sorted result.

Default run (13 steps): SELECT title, plays FROM track WHERE plays >= 95 AND rating >= 4 ORDER BY plays DESC LIMIT 2. FROM: start with all 8 rows of track. … Result: 2 rows.

Simplified: A toy relational engine in the browser, not SQLite itself: a small structured query model built from the controls (no SQL text is parsed or executed), typed INTEGER/TEXT/NULL values, and tables of at most a few rows. Results were checked against Python's sqlite3 (SQLite 3.49). Ties in ORDER BY are broken by id here (by the selected columns with DISTINCT), and DISTINCT without ORDER BY keeps first-occurrence order; SQLite itself promises no order for ties. LIKE is case-insensitive for ASCII letters, as in SQLite.

Educational simulation

Loading the simulation…

Where Should the Scaler Be Fitted?

Hyperparameter tuning with GridSearchCV is usually introduced as a simple loop. You give it an estimator, a grid of candidate settings, and a number of cross-validation folds, and it reports which combination scored best. The part that is easy to miss is what happens to the data before that loop starts. Many workflows scale or transform the features first, and then hand the transformed data to the search. That ordering feels natural, and it is the source of one of the most common errors in tuning.

What do you think happens?

Before any cross-validation folds are formed, should a scaler that learns its mean and standard deviation from the data be fitted on the whole dataset, or only on the part that each fold uses for training?

On the whole dataset, once, before splitting
Only on the training portion, inside each fold

Reveal answer

Answer:

Only on the training portion, inside each fold.

The source states that the splitting of the dataset during cross-validation should happen before any preprocessing, and that any process that extracts knowledge from the dataset should only ever be applied to the training portion. A scaler learns statistics from data, so it falls under that rule.

The Belief Behind the Leak

The mistake starts from a belief that seems reasonable: preprocessing is just a data-preparation step, so it can be done once, on everything, before any modeling begins. Cross-validation then looks like a fair test, because each fold is a different split of data that has already been prepared. The reasoning is that the validation portion is only being used for scoring, so it is harmless if the preprocessing has already seen it.

The belief breaks down because a scaler is not a neutral formatting step. Standardizing features means computing statistics such as the mean and standard deviation from the data and then using them to transform values. Once the scaler has computed those statistics from every row, the validation rows have already influenced the numbers the model will train on. The validation portion is no longer unseen when it is scored.

Tracing the Leak Step by Step

Worked example

How a whole-dataset mean leaks validation information

Six values of one feature are split into a training portion of four rows and a validation portion of two rows. Compare the mean the scaler would learn when it is fitted on everything with the mean it would learn from the training portion alone.

1
List the values
The training portion holds 2, 4, 6, and 8. The validation portion holds 10 and 12.
2
Compute the mean when the scaler sees everything
The sum of all six values is 42, so the mean is $\bar{x}_{\text{all}} = \frac{42}{6} = 7$. This is the number a leaky scaler would use to transform the training rows.
3
Compute the mean when the scaler sees only training rows
The sum of the training values is 20, so the mean is $\bar{x}_{\text{train}} = \frac{20}{4} = 5$. This is the number a correct scaler would learn inside this fold.
4
Identify where the leak enters
The gap between 7 and 5 comes entirely from the validation values 10 and 12. The training computation has absorbed information from rows that are supposed to be unseen.

Result

A scaler fitted on the whole dataset learns a mean of 7 instead of 5, so the validation rows have shaped the training transformation before any scoring happens.

The same idea applies to any step that extracts knowledge from the data, not only scalers. The source's rule is that such a process should only ever be applied to the training portion of the dataset. The leak happens because the statistic is learned from rows that the model is later judged on, which makes the resulting score look better than the model would perform on genuinely new data.

Cross-Validation as the Outer Loop

The source gives a direct rule for this problem. The splitting of the dataset during cross-validation should happen before any preprocessing. That makes cross-validation the outermost loop in the processing workflow. Read as a nested structure, the outer loop decides which rows are for training and which are for validation, and everything inside the loop must respect that decision.

What contains what: the fold split sits on the outside, and the step that learns from data sits inside the training portion of each fold, so each fold's statistics come only from that fold's training rows.

This is the reason the word outermost matters. The loop that splits the data sits on the outside, and anything that learns from data sits inside it. In the diagram, the validation portion has no arrow leading to the scaler, which is exactly the property the correct workflow must have.

Letting a Pipeline Enforce the Order

Writing the correct order by hand inside a loop is error-prone. A Pipeline solves this by bundling the preprocessing step and the estimator into one object. The source notes that GridSearchCV can be used with a pipeline, which means the search can treat the whole bundle as its estimator. When that happens, the preprocessing is refitted inside each fold automatically, so the scaler only ever learns from the training portion of that fold.

python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

pipe = Pipeline([("scaler", StandardScaler()), ("svc", SVC())])
param_grid = {"svc__C": [0.1, 1, 10]}
search = GridSearchCV(pipe, param_grid, cv=5)
search.fit(X, y)

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

pipe = Pipeline([("scaler", StandardScaler()), ("svc", SVC())])
param_grid = {"svc__C": [0.1, 1, 10]}
search = GridSearchCV(pipe, param_grid, cv=5)
search.fit(X, y)

Notes on the code

The Pipeline bundles the scaler and the estimator into one object.
The grid key svc__C uses the step name svc, followed by a double underscore and the hyperparameter name C.
GridSearchCV receives the pipeline as its estimator, so each fold refits the scaler on that fold's training rows.
X and y are assumed to be the feature matrix and target vector already loaded.

The parameter grid must name each hyperparameter with its step prefix. Because the pipeline has a step called svc, the regularization setting C is written as svc__C. Without the prefix, the search would not know which step the candidate value belongs to.

For each fold and each hyperparameter candidate, the search asks the pipeline to fit on training rows, the scaler learns from those training rows only, and the scaler then transforms the validation rows using that learned statistic.

Comparing the Leaky and Correct Workflows

The two approaches differ in one place: where the preprocessing sits relative to the split. The diagram below sets them side by side so you can check each property against the rule the source gives.

In the leaky workflow the scaler is fitted before the split, so the validation rows shape the statistics. In the correct workflow the split comes first and the scaler is refitted on each fold's training rows.

Common mistake
Scaling the full feature matrix before calling GridSearchCV
What it looks like
```
X_scaled = StandardScaler().fit_transform(X), then search.fit(X_scaled, y)
```
Why it fails
The scaler's statistics are computed from every row, including the rows each fold will later treat as validation data.
Do this instead
Place the scaler inside a Pipeline and pass the pipeline to GridSearchCV, so it is refitted on each fold's training rows.
Common mistake
Writing an unprefixed parameter name in the grid
What it looks like
```
param_grid = {"C": [0.1, 1, 10]} for a pipeline with a step named svc
```
Why it fails
A pipeline's hyperparameters are addressed through their step names, so the search needs to know which step the value belongs to.
Do this instead
Use the step prefix, such as svc__C.
Common mistake
Treating a cross-validation score as trustworthy without checking where the preprocessing was fitted

Finding the Step Where Results Go Wrong

When a tuning result disappoints or seems too good, the cause is often a specific step rather than the model itself. Work through the workflow in order: where is the split, which steps run inside each fold, and which data does each step see? Locating the point where a learned statistic came from validation data is usually the fastest route to the source of an unexpectedly high or unstable score.

MEDIUM

A colleague reports that a tuned model scores much higher in cross-validation than on a fresh test set. Their code scales the full feature matrix with a StandardScaler, then passes the scaled matrix to GridSearchCV. Identify the step where the score first becomes optimistic, and describe how you would restructure the code so the scaler is refitted inside each fold.

Hints

Ask which rows the scaler's mean was computed from at the moment it was fitted.
Think about whether the scaler is inside or outside the loop that splits the data into folds.
A Pipeline containing the scaler and the estimator, passed to GridSearchCV, puts the preprocessing inside each fold.

Your attempt (not saved)

Key Takeaways

Preprocessing that learns from data, such as scaling, must only see the training portion of each cross-validation fold.
Cross-validation should be the outermost loop, so the split happens before any preprocessing.
A Pipeline passed to GridSearchCV refits the preprocessing inside every fold, which enforces the correct order automatically.
Pipeline hyperparameters in a grid must be named with their step prefix, such as svc__C.
When a tuning score looks too good, locate the step where a learned statistic was computed from validation rows.

Related concepts

More concepts starting with P