The Problem with Preprocessing Outside Cross-Validation
A leak is defined by where information crosses, not by the presence of a particular algorithm. If something learned from the held-out portion influences the training side, the boundary between the two has been crossed.
Try it: K-Fold Cross-Validation and Model Selection
How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.
How it works
- Set 10 test points aside; they take no part in any choice.
- Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
- For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
- Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
- Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
- For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.
Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.
Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.
Loading the simulation…
Why a Good Score Can Mislead
Imagine you build a model, run cross-validation, and see a strong result. That result is only useful if the splits used during cross-validation behave like the situation the model will face later. Cross-validation exists to estimate how a model will behave on data it has not seen. When that assumption holds, the score tells you something trustworthy. When it breaks, the number on your screen can flatter the model, and you may make decisions based on it without realizing anything went wrong.
This article looks at one specific way the assumption breaks: preprocessing that is performed outside the cross-validation loop. The structure of cross-validation can look perfectly correct while information still crosses from the held-out part of the data into the modeling process. The order of operations decides whether a score is fair, so it matters as much as the splits themselves.
Tracing Where the Held-Out Rows Go
The easiest way to see the problem is to follow the rows step by step. Consider an illustrative scenario, written for this article rather than taken from the source. A preprocessing step, such as a scaler that learns the average and spread of a numeric column, is fit on the full dataset first. Only afterward is the data split into folds for cross-validation. Watch where the information from the held-out rows travels.
Tracing one leaked workflow
A dataset has a numeric column. The workflow scales that column using statistics computed on all rows, then runs cross-validation on the scaled data. Trace what information reaches the training step.
- 1Step 1: Compute statistics
The scaler reads every row, including rows that will later be held out, and records an average and a spread.
- 2Step 2: Transform the data
Every row is scaled with those shared numbers, so the training rows are already shaped by values from the held-out rows.
- 3Step 3: Split and train
Cross-validation splits the scaled data and trains the model on the training portion of each fold.
- 4Step 4: Score
The held-out portion is scored. Its values helped set the scaling, so the score is computed on a setup that has already been contaminated.
The cross-validation loop looks correct, but information from the held-out rows reached the modeling process before the split happened. The resulting score is no longer a fair test of unseen data.
What Data Leakage Means
Data leakage occurs when information from parts of the data is leaked into the modeling process. Once that happens, the evaluation you run is no longer a fair test of how the model would perform on unseen data.
The key word is information. A leak is not a missing value or a bad column by itself. It is any signal that should have stayed on the held-out side of the split but instead influenced how the model was built or prepared. In the scenario above, the held-out rows never entered the model directly, yet their values helped determine the scaling, which is enough to carry information across the boundary.
A leak is defined by where information crosses, not by the presence of a particular algorithm. If something learned from the held-out portion influences the training side, the boundary between the two has been crossed.
How Leakage Breaks the Resemblance to New Data
Cross-validation is meant to imitate the situation the model will face later, when it meets data it has never seen. Each fold is supposed to play the role of new data. The source describes the consequence directly: when leakage happens during cross-validation, the splits no longer correctly mirror how new data will look to the modeling process. The structure of the folds is still there, but the information that should have stayed separated has crossed over.
| Aspect | Fold that sees only its own training data | Fold preprocessed using all rows |
|---|---|---|
| What shapes the preprocessing | Only the training portion of that fold | Every row, including the held-out portion |
| Resemblance to unseen data | Held-out rows are genuinely new when the model is prepared | Held-out rows were already part of the preparation, so they are not new |
| Meaning of the score | A fair estimate of performance on unseen data | An estimate computed on a contaminated setup |
Why a leaked fold no longer stands in for new data
The two columns differ in a single respect that matters a great deal. In the fair version, the held-out rows are truly unseen at the moment the model is prepared. In the leaked version, they have already influenced the preparation, so the fold no longer resembles data arriving later.
Two Consequences of Leakage
The source identifies two main consequences of leakage. The first is overly optimistic results during cross-validation. Because the held-out portion has already shaped the model's inputs, the score tends to flatter the model, and the number reported is higher than the model would achieve on genuinely new data.
The second consequence is the possible selection of suboptimal parameters. Cross-validation is often used to compare settings and choose the best one. If every candidate setting is judged through contaminated folds, the comparison itself can point to the wrong choice. The parameter that looks best may be the one that benefits most from the leak, not the one that will generalize best.
Following Each Fold in the Correct Order
A fair workflow keeps the preprocessing inside the loop. In each iteration, the preprocessing is fit only on the training portion of that fold, and the same fitted preprocessing is then applied to the held-out portion. Trace the steps of one iteration to see how the boundary is kept intact.
Compare this with the leaked workflow traced earlier. The difference is not the number of steps. It is the place where the statistics are computed. In the fair version, the fit happens inside the iteration and only ever sees the training portion, so the held-out portion remains new when the score is produced.
Mistakes That Hide a Leak
Common mistake Preprocessing the full dataset before calling cross-validation
What it looks likeScaling every row first, then passing the scaled table into the cross-validation routine.
Why it failsThe held-out rows have already influenced the scaling, so the folds no longer mirror how new data will look.
Do this insteadPlace the preprocessing inside the loop and fit it only on each training portion.
Common mistake Judging a score only by whether the splits look correct
What it looks likeChecking that the folds have the right sizes and then trusting a strong result without asking when the preprocessing ran.
Why it failsA leak can hide inside a workflow that appears to use proper splits, so the order of operations matters when you judge a score.
Do this insteadTrace where each fitted step gets its numbers and confirm that nothing learned from the held-out portion reaches the training side.
Common mistake Assuming a high score means the model is good
What it looks likeChoosing a parameter because it produced the best cross-validation number.
Why it failsLeakage causes overly optimistic results and can push the selection toward suboptimal parameters.
Do this insteadVerify the evaluation setup first, then compare candidate settings on fair folds.
Common mistake Believing the leak only matters for complicated models
Practice Tracing a Workflow
A colleague fills missing values in a column using the average computed from the entire dataset, then runs cross-validation on the filled table. Trace which rows shaped the fill value, decide whether the held-out folds still resemble new data, and describe how you would restructure the workflow.
Hints
- Ask which rows were read when the average was computed.
- Ask whether a held-out row could have influenced a value that the training side later uses.
- The fix is to move the fill step inside each iteration and compute the average only from the training portion of that fold.
Key Takeaways
- Data leakage occurs when information from parts of the data is leaked into the modeling process, so the evaluation stops being a fair test of performance on unseen data.
- Preprocessing fit on all rows before splitting lets held-out information reach the training side, and the splits no longer mirror how new data will look.
- The two main consequences of leakage are overly optimistic cross-validation results and the possible selection of suboptimal parameters.
- A leak can hide inside a workflow that appears to use proper splits, so trace the order of operations before trusting a score.
- Keep preprocessing inside each cross-validation iteration and fit it only on the training portion of that fold.