Mastering Model Evaluation and Selection
Evaluation data must be kept separate from the data used to choose a model, because reusing it produces overly optimistic estimates.
Try it: K-Fold Cross-Validation and Model Selection
How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.
How it works
- Set 10 test points aside; they take no part in any choice.
- Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
- For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
- Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
- Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
- For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.
Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.
Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.
Loading the simulation…
Why One Dataset Cannot Do Three Jobs
Imagine you have built several candidate models for the same prediction problem. Each one is a different model, or the same model with different settings. You want to know which one to deploy, and you want a number that tells you how well it will perform on data it has never seen. It is tempting to use a single dataset for all of this: fit each candidate, compare their scores on the data you held back, keep the winner, and report its score as the expected result. This article explains why that tempting routine gives an answer that looks better than it really is, and how to arrange your data so the final number can be trusted.
What do you think happens?
You score five candidate models on a held-out set, keep the one with the highest score, and report that score as how well the model will do on new data. Is that reported score an honest estimate?
Reveal answer
No. The held-out portion was used twice: first to choose the model, and then to judge it. Its score is therefore biased upward.
The comparison step that feels reasonable is the one that spends the held-out data on selection. Once the data has shaped the choice, it can no longer stand in for unseen future data.
Tracing One Model Choice from Split to Report
A standard split divides the available data into three roles. Training data is used to build the model. Validation data is used to choose among models and to pick parameter values. Test data is used for the final evaluation. The order of work matters. You split first, so the test data is set aside before any comparison begins. Then you fit the candidates on the training data, compare them on the validation data, and make one choice. Only after that choice is finished do you open the test data, once, to measure how the chosen model is likely to behave on future data.
Read the diagram as a strict sequence. The test set is set aside at the start and then left alone. The validation set is the only place where candidates are compared against each other. Once one model has been chosen, the test set is used a single time. Each step consumes only the data its role allows.
Three Roles for Three Sets
Each set has a single job, and keeping those jobs separate is what protects the final estimate. The training set is where the model learns its internal values. The validation set is where you compare candidate models and adjust settings, such as how strongly a model is regularized. The test set is touched once, at the end, to measure how the chosen model is likely to behave on future data.
Splitting a Dataset in Python
The following example uses a plain Python list of ten numbered examples to stand in for a dataset. Each slice takes a different portion of the list, so the three sets never overlap. In a real project the same idea applies to your rows of data. Libraries provide shuffling and stratified splitting, which this example leaves out, so the split here simply takes the first six items for training, the next two for validation, and the last two for testing.
train: [1, 2, 3, 4, 5, 6]
validation: [7, 8]
test: [9, 10]
overlap check: set()- data is the stand-in dataset of ten numbered examples
- the slices use half-open ranges, so 0:6 takes indices 0 through 5
- the test slice is written once and is not used anywhere in selection
Tracing where each item goes
Given the ten numbered items, which role does item 7 play, and which role does item 9 play?
- 1Locate item 7
Item 7 sits at index 6 in the list, because indexing starts at zero.
- 2Match index 6 to a slice
The validation slice data[6:8] starts at index 6, so item 7 belongs to validation.
- 3Locate item 9
Item 9 sits at index 8.
- 4Match index 8 to a slice
The test slice data[8:10] starts at index 8, so item 9 belongs to test. It must not be used to pick a model.
Item 7 is a validation example and item 9 is a test example. The two never overlap, so the choice made on validation cannot leak into the test score.
How Reusing the Test Set Inflates the Score
Now trace what happens when the test set is consulted during selection. Suppose five candidate models are each scored on the same held-out portion, and the best score is kept and reported. The held-out data has now been used twice. In the first use it decided which model to keep. In the second use it was supposed to judge that model. The test data has been used up, because the model has been tuned to it, so it can no longer stand in for unseen future data.
Following the test set through a selection loop
Five candidate models are scored on the test set: 0.81, 0.84, 0.79, 0.86, and 0.83. The team keeps the best and reports its score as the expected future performance.
- 1Compare candidates on test
The test set is now the basis for picking a winner, not only for judging one. This is the step that spends the test data on selection.
- 2Keep the highest score
The model scoring 0.86 is kept because it has the highest test score among the five.
- 3Report the same score
The reported figure of 0.86 comes from the same data that chose the model, so it is a biased upward estimate rather than an independent one.
- 4Judge what the number means
Because the choice was shaped by the test data, the 0.86 overstates how the model will perform on new data. An honest estimate would require data untouched by selection.
The reported 0.86 is optimistic. The test set was consulted to decide which model to keep, so it can no longer provide an unbiased estimate of future behavior.
The same trap appears if cross-validation is used to make selection decisions on the data you plan to report. Any procedure that looks at an evaluation score and then chooses something based on that score leaves the score biased. This is why the test set should be consulted only after all selection decisions are finished.
Cross-Validation as a Replacement for the Fixed Validation Split
A single split is not the only option. Instead of setting aside one fixed validation portion, each role can be handled with cross-validation. The most commonly used arrangement keeps a training and test split for the final evaluation, and then applies cross-validation on the training portion to make model and parameter choices. The test set remains untouched by selection in this arrangement.
Cross-validation works by training and testing on several different subsets of the available data, so that each portion takes a turn being held out for checking. Because every portion is eventually used for checking, the choice of model does not depend on one lucky or unlucky split. The choices are made using training-side data, and the final number comes from data that was not involved in those choices. Both approaches aim at the same goal: choosing a model without spending the data you need for an honest evaluation. They differ in how the training and checking roles are arranged across the data.
round 1 validate: [1, 2] train: [3, 4, 5, 6]
round 2 validate: [3, 4] train: [1, 2, 5, 6]
round 3 validate: [5, 6] train: [1, 2, 3, 4]- train is only the training portion, so the test set is never part of the folds
- each fold takes a turn being held out for checking
- every item is used for checking exactly once across the three rounds
Mistakes Beginners Make with Held-Out Data
Common mistake Comparing candidate models on the test set and reporting the best score
What it looks likeFive models are scored on the test set, the highest score is kept, and that same number is reported as expected performance.
Why it failsThe test data shaped the choice, so it is no longer an independent judge of the chosen model.
Do this insteadCompare candidates on the validation set or with cross-validation on the training portion, and touch the test set once after all choices are made.
Common mistake Tuning a setting while watching the test score
What it looks likeA regularization strength is nudged up and down until the test score looks best.
Why it failsAdjusting settings is a selection decision. Doing it against the test data uses up the test data.
Do this insteadAdjust settings against the validation data or the cross-validation results, then evaluate once on test.
Common mistake Letting the sets overlap
What it looks likeA slice such as data[5:9] is used for validation while data[8:10] is used for testing.
Why it failsItems in both sets make the validation choice and the test judgement depend on the same examples.
Do this insteadGive each role its own non-overlapping portion, as the slices in the example do, and check for overlap explicitly.
Common mistake Treating one lucky split as the final answer when a single fixed validation portion can be unrepresentative
Checking Your Understanding of Held-Out Data
A colleague says: I tried four models on the test set, picked the one with the best test score, and reported that score. Explain in your own words why the reported number is optimistic, and describe the order of steps you would follow instead.
Hints
- Ask how many times the test data was used, and for what purpose.
- Think about which role the validation set or cross-validation should play in the choice.
- The test set should be consulted only after all selection decisions are finished.
Key Takeaways
- A standard split gives each set one job: training data builds the model, validation data chooses models and parameters, and test data provides the final evaluation.
- Reusing the test set for selection uses it twice, so the reported score is biased upward and no longer estimates performance on unseen data.
- Cross-validation can replace a fixed validation split by rotating which portion is held out, with selection done on training-side data and the test set kept untouched.
- The central rule is that the data used to choose is not the data used to judge, and the test set should be consulted only after all selection decisions are finished.