Cross-Validation for Model Selection
The data used to choose is not the data used to judge. Selection decisions and the final score must come from different portions of the data.
Try it: K-Fold Cross-Validation and Model Selection
How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.
How it works
- Set 10 test points aside; they take no part in any choice.
- Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
- For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
- Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
- Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
- For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.
Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.
Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.
Loading the simulation…
A Tempting Shortcut With Hidden Bias
Suppose you have a dataset and several candidate models, or several settings of the same model. A natural first attempt goes like this. You split your data once, train each candidate on the training portion, and compare them on the held-out portion. You keep the one with the best score and report that score as how well the model will perform on new data. Each step seems reasonable, and the comparison on the held-out portion is the step that feels most natural. The problem is hidden in that step: the held-out portion is being used twice. First it helps choose the model, and then it is used to judge the model that was chosen.
What do you think happens?
You compare three candidate models on a held-out portion, keep the best one, and report its score on that same portion. Is the reported score an honest estimate of performance on new data?
Reveal answer
No. The held-out portion was used twice, first to choose the model and then to judge it, so the reported score is biased in an optimistic direction.
The purpose of an evaluation set is to stand in for unseen future data. Once that data has shaped which model was kept, it no longer plays that role, and the score it produces is inflated.
Three Jobs, Three Datasets
A standard split divides the available data into three subsets, and each one has a single job. The training set is where the model learns its internal values. The validation set is where you compare candidate models and adjust settings, such as how strongly a model is regularized. The test set is touched once, at the end, to measure how the chosen model is likely to behave on future data. Keeping these jobs separate is what protects the test estimate.
| Subset | Main job | When it is used |
|---|---|---|
| Training set | The model learns its internal values | Throughout model fitting |
| Validation set | Compare candidate models and adjust settings | During selection, repeatedly |
| Test set | Estimate how the chosen model will behave on future data | Once, at the very end |
Each subset has one purpose and a fixed place in the workflow.
The data used to choose is not the data used to judge. Selection decisions and the final score must come from different portions of the data.
How the Test Set Gets Used Up
The purpose of a test set is to estimate how a model will perform in the future. That estimate is trustworthy only if the test data has played no part in shaping the model. Imagine you train several candidates and then compare their test scores, keeping the one that scores highest. The test data has now helped decide which model to keep. The model has been tuned to it, so the test data can no longer stand in for unseen future data. Every further comparison on the test set makes the final number a little more optimistic than the real performance you should expect.
Splitting Ten Examples in Python
To see the three roles in code, imagine a dataset of ten numbered examples. A plain Python list stands in for the data here. Each slice takes a different portion of the list, so the three subsets never overlap. The training slice comes first, the validation slice is used for comparing candidates, and the test slice is set aside until the very end. In a real project the same idea applies to rows of data, and libraries provide shuffling and stratified splitting, which this small example leaves out.
[1, 2, 3, 4, 5, 6]
[7, 8]
[9, 10]- examples holds the ten numbered examples that stand in for a dataset
- train takes the first six examples, which the model learns from
- validation takes examples seven and eight, used to compare candidates and settings
- test takes the remaining examples nine and ten, to be consulted only once at the end
Notice the order of use. The training slice is used to fit each candidate. The validation slice is used to compare those candidates and choose a model or a setting. Only after every selection decision is finished should the test slice be used to produce the final number. If you find yourself going back to the test slice to pick between two options, the split has stopped doing its job.
Rotating Validation Through Folds
A single validation split has a weakness: the result depends on which examples happened to land in it. Cross-validation addresses this by training and testing on several different subsets of the available data, so that each portion takes a turn being held out for checking. Because every portion is eventually used for checking, the choice of model does not depend on one lucky or unlucky split. The choices are still made using training-side data, and the final number still comes from data that was not involved in those choices.
Mistakes When Choosing Models
Common mistake Reporting the best held-out score as the expected performance
What it looks likeThree candidates are compared on one held-out portion, the highest score wins, and that same score is reported as the estimate.
Why it failsThe held-out portion helped choose the winner, so the score is biased upward.
Do this insteadKeep a separate test portion for the final estimate and use validation or cross-validation only for selection.
Common mistake Checking the test set after each tuning step
What it looks likeA regularization strength is tried, the test score is read, a different value is tried, and the test score is read again.
Why it failsEach look at the test data lets it shape the model, so it stops standing in for unseen future data.
Do this insteadMake every selection decision on validation data, and consult the test set once after all decisions are finished.
Common mistake Assuming cross-validation makes selection leak-proof
Why it failsCross-validation on the data you plan to report still uses an evaluation score to make choices, which biases that score.
Do this insteadRun cross-validation on the training portion only, and report the score from the untouched test set.
Common mistake Building the three subsets without deciding the order of use first
Checking Your Own Split Plan
You have 100 labeled examples and two candidate settings for a model. Describe which examples will be used to train each setting, which will be used to choose between them, and which will be held back for the final score. Then state at what point in the workflow the held-back examples may be touched.
Hints
- Recall the single job of each subset: training, validation, and test.
- The held-back set must not influence which setting you keep.
- Think about whether cross-validation on the training portion could replace the fixed validation portion, and where the test portion sits in that arrangement.
Key Takeaways
- A model needs separate data for learning, for choosing between candidates, and for a final honest evaluation.
- Reusing the test set to choose a model or parameter makes its score optimistic, because the test data has shaped the model.
- Cross-validation can replace a single validation split by rotating the held-out fold, while a test split still provides the final evaluation.
- The data used to choose is not the data used to judge, and the test set should be consulted only after all selection decisions are finished.