Concepts / Cross-Validation and Data Splitting

Cross-Validation and Data Splitting

Preprocessing that learns from data, such as scaling, must only see the training portion of each cross-validation fold.

  • Programming
Interactive lab

Try it: SELECT, WHERE, ORDER BY, LIMIT

How a SELECT query tests every row against the WHERE condition (keeping only rows where it is TRUE, not FALSE or NULL), then picks columns, removes duplicates with DISTINCT, sorts with ORDER BY and cuts with LIMIT.

How it works

  1. FROM: start with every row of the table.
  2. WHERE: evaluate the condition on each row. Comparisons with NULL give NULL (unknown); NOT, AND and OR follow three-valued logic, with AND binding tighter than OR. Only rows where the whole condition is TRUE are kept.
  3. SELECT picks the listed columns; DISTINCT then drops result rows that repeat an earlier one.
  4. ORDER BY sorts by each key in turn, ASC or DESC; SQLite puts NULL first when ascending and last when descending.
  5. LIMIT n keeps only the first n rows of the sorted result.

Default run (13 steps): SELECT title, plays FROM track WHERE plays >= 95 AND rating >= 4 ORDER BY plays DESC LIMIT 2. FROM: start with all 8 rows of track. … Result: 2 rows.

Simplified: A toy relational engine in the browser, not SQLite itself: a small structured query model built from the controls (no SQL text is parsed or executed), typed INTEGER/TEXT/NULL values, and tables of at most a few rows. Results were checked against Python's sqlite3 (SQLite 3.49). Ties in ORDER BY are broken by id here (by the selected columns with DISTINCT), and DISTINCT without ORDER BY keeps first-occurrence order; SQLite itself promises no order for ties. LIKE is case-insensitive for ASCII letters, as in SQLite.

Educational simulation

Loading the simulation…

Why a Tuning Score Can Mislead

Hyperparameter tuning with GridSearchCV is usually introduced as a simple loop. You give it an estimator, a grid of candidate settings, and a number of cross-validation folds, and it reports which combination scored best. The part that is easy to miss is what happens to the data before that loop starts. Many workflows scale or transform the features first and then hand the transformed data to the search. That ordering feels natural, and it is the source of one of the most common errors in tuning.

This article follows one workflow step by step. You will watch where the data goes, what each cross-validation fold is allowed to learn from, and where a score can stop meaning what it appears to mean. Keep one question in mind as you read: at which step does the learner's expectation about the data first stop matching what the code actually does?

What do you think happens?

Preprocessing is done once on all rows, then cross-validation runs on the prepared data. Does this workflow produce an honest estimate of how the model will perform on new data?

  • Yes, because every fold is a different split of the data

  • No, because the validation rows have already influenced the preprocessing

Reveal answer
Answer:

No, because the validation rows have already influenced the preprocessing

The misconception is that the validation portion is only used for scoring, so it is harmless if preprocessing has already seen it. But a scaler that learns its statistics from all rows has learned something from the validation rows before scoring begins, so the reported score can be misleading.

Tracing Where the Data Goes

Preprocessing that learns from data, such as scaling, must only see the training portion of each cross-validation fold. A standard scaler works by computing statistics, such as the mean and standard deviation of each feature, and then using them to transform the values. The question is which rows those statistics come from. If the scaler is fitted on the whole dataset, the statistics include the rows that a fold will later treat as validation rows.

The diagram below traces the same raw data through two workflows. In the flawed workflow, the scaler learns from every row before the split exists. In the correct workflow, the split happens first and the scaler learns only from the training rows of each fold.

All rowsScaler fit on allrowsScaled featuresCross-validationfoldsValidation rowsscoredAll rowsSplit into foldsScaler fit ontraining rowsValidation rowstransformed onlyValidation rowsscoredstatistics computedtransformsplit after scalingscoresplit firsttraining rows onlyapply learned statisticsscore
Which rows does the scaler's mean and standard deviation come from in each workflow, and where does validation information enter the training process?

In the leaky branch, the validation rows have already shaped the numbers the model receives. The validation score is no longer a clean test of how the fitted workflow handles rows it has never learned from. In the correct branch, the validation rows are only transformed using statistics computed from the training rows, so they remain unseen by the learning step.

Why the Split Must Come First

The source gives a direct rule for this problem. The splitting of the dataset during cross-validation should happen before any preprocessing. Any process that extracts knowledge from the dataset should only ever be applied to the training portion of the dataset. That makes cross-validation the outermost loop in your processing.

Read as a nested structure, the outer loop decides which rows are for training and which are for validation, and everything inside the loop must respect that decision. Preprocessing that learns from data is one of those things. It sits inside the loop, not in front of it.

Cross-validationloopTraining rows of foldPreprocessing fitValidation rows offoldEstimator fitScore on held-outrowscontainscontainslearns fromlearns fromscored on
What contains what: is preprocessing nested inside each fold, or is cross-validation the outer container around the whole pipeline?
PropertyPreprocessing before the splitPreprocessing inside each fold
Where preprocessing sitsBefore the cross-validation loopInside the loop, after the split
Which rows the scaler learns fromAll rows, including future validation rowsOnly the training rows of the current fold
What validation rows experienceAlready shaped by statistics from themselvesTransformed using statistics from training rows only
Reliability of the reported scoreCan be optimistic or unstableHonest estimate of performance on unseen rows

The two approaches differ in one place: where the preprocessing sits relative to the split.

Building the Pipeline Inside GridSearchCV

Writing the order by hand inside a loop is error-prone. A Pipeline solves this by bundling the preprocessing step and the estimator into one object. The source notes that GridSearchCV can be used with a pipeline, which means the search can treat the whole bundle as its estimator. When that happens, the preprocessing is refitted inside each fold automatically, so the scaler only ever learns from the training portion of that fold.

python
Notes on the code
  • The Pipeline bundles the scaler and the SVC into one estimator.
  • The parameter grid names C with its step prefix svc__C, because C belongs to the svc step.
  • GridSearchCV receives the whole pipeline, so the split happens before any scaling.
python
Notes on the code
  • The scaler is fitted on every row before GridSearchCV sees the data. This is the flawed ordering, shown only so you can recognize it.
  • Because the scaling happened first, no fold can refit the scaler on its own training rows.
GridSearchCVPipelineScalerSVCHeld-out fold1. fit pipeline on training rows of fold2. fit scaler on training rows only3. fit SVC on scaled training rows4. score pipeline on held-out rows
For each fold and each hyperparameter setting, what happens next and in what order: fit the scaler, fit the model, then score on the held-out fold?

Read the sequence top to bottom for one fold and one candidate setting. The search asks the pipeline to fit on the training rows of that fold, the scaler learns its statistics from those rows, the SVC fits on the transformed rows, and only then is the held-out fold scored. The search then repeats this for the next fold and the next setting.

Tracking Which Rows Each Fold Uses

Each cross-validation fold is a different partition of the same rows. In one fold, a set of rows serves as validation; the remaining rows form the training portion. The index sets change from fold to fold, so a row that is validation in one fold is training in another. The scaler inside the pipeline must be refitted for every fold because the training portion changes each time.

Fold 1validation rows 0 and 1;training rows 2 to 5Fold 2validation rows 2 and 3;training rows 0, 1, 4, 5Fold 3validation rows 4 and 5;training rows 0 to 3
Which samples are training and which are validation in each fold, and how do those index sets change from fold to fold?
Worked example

What the scaler sees in fold 2

Six rows are split into three folds. In fold 2, which rows may the scaler learn from, and which rows may it only transform?

  1. 1
    Identify the validation rows

    In fold 2, rows 2 and 3 are the validation rows for this fold.

  2. 2
    Identify the training rows

    The training rows are the remaining rows, 0, 1, 4 and 5.

  3. 3
    Fit the scaler

    Inside the pipeline, the scaler computes its statistics only from rows 0, 1, 4 and 5.

  4. 4
    Transform the validation rows

    Rows 2 and 3 are passed through the already-fitted scaler and scored. They do not contribute to the statistics.

Result

The scaler learns from rows 0, 1, 4 and 5 only. Rows 2 and 3 are used only for scoring, which keeps the fold's validation score honest.

Mistakes That Break the Fold Boundary

  • Common mistake

    Scaling the full dataset before calling GridSearchCV

    What it looks like
    X_scaled = StandardScaler().fit_transform(X) followed by a search fitted on X_scaled
    Why it fails

    The scaler's statistics include the rows that each fold later uses for validation, so the validation portion has already influenced training.

    Do this instead

    Put the scaler inside a Pipeline and pass the pipeline to GridSearchCV, so it is refitted on the training portion of each fold.

  • Common mistake

    Assuming the validation rows are harmless because they only get scored

    Why it fails

    Scoring is the final step. Any learning that happened earlier, such as computing a mean from those rows, already used them.

  • Common mistake

    Writing an unprefixed hyperparameter name for a pipeline

    What it looks like
    param_grid = {"C": [0.1, 1, 10]} with a pipeline that has an svc step
    Why it fails

    A pipeline has several steps, and the bare name does not say which step owns the setting.

    Do this instead

    Use the step prefix, such as svc__C.

  • Common mistake

    Placing the split after preprocessing because the data already looks clean

    Why it fails

    Cleanliness is not the criterion. The rule is that the split must happen before any step that extracts knowledge from the data.

    Do this instead

    Make the split the outer container and keep learning steps inside it.

Finding Where a Result Drifts

When a tuning result disappoints or seems too good, the cause is often a specific step rather than the model itself. Work through the workflow in order. Where is the split? Which steps run inside each fold? Which data does each step see? Locating the point where a learned statistic came from validation data is usually the fastest route to the source of an unexpectedly high or unstable score.

MEDIUM

A colleague reports a suspiciously high cross-validation score. Their workflow loads the data, fits a scaler on all rows, then runs GridSearchCV on the scaled data. Starting from the raw data, at which step does the reported score first stop matching what the learner expected, and what single change would fix it?

Hints
  • Trace the data from the raw rows forward and note the first step that learns a statistic.
  • Ask whether that step can see the validation rows of any fold.
  • The fix moves the split ahead of the learning step, usually by wrapping the scaler and estimator in a Pipeline.

Key Takeaways in Practice

  1. 1
    Preprocessing that learns from data, such as scaling, must only see the training portion of each cross-validation fold.
  2. 2
    Cross-validation is the outermost loop: the split happens first, and anything that learns from data sits inside it.
  3. 3
    A Pipeline passed to GridSearchCV refits the preprocessing inside every fold, which enforces the correct order automatically.
  4. 4
    Name pipeline hyperparameters with their step prefix, such as svc__C, and when a result looks wrong, trace back to the first step that learned from validation rows.

At a glance

  • Preprocessing that learns from data must only see the training portion of each cross-validation fold.
  • Cross-validation should be the outermost loop, so the split happens before any preprocessing.
  • A Pipeline passed to GridSearchCV refits the preprocessing inside every fold, enforcing the correct order automatically.
  • Pipeline hyperparameters are named with their step prefix, such as svc__C.
  • When a tuning result looks too good or unstable, locate the first step that learned from validation data.

Related Topics

Related concepts

More concepts starting with C