Training and Test Splits in Machine Learning
Hyperparameters are settings chosen before training, and a validation dataset is the held-out data used to find the best of them.
Why Settings Need Their Own Data
When you train a neural network, you make choices that the training process does not make for you. The learning rate, the number of training epochs, and the size of each batch are settings you pick before training starts. Finding good values for these settings is a central part of building a working model. This article answers a practical question: which data should you use to judge those choices, and why the answer is not always the dataset you expect.
Many learners start with a reasonable-sounding plan. You already have a test set that the model never sees during training, so you use it to decide which settings work best. You try a learning rate, check the score on the test set, try another value, check again, and keep whichever scores highest. This feels sensible because it treats every held-out split as one generic kind of data that can be consulted as often as needed.
What do you think happens?
You try ten learning rates, check the test set score each time, and report the best one. Is that reported score an honest estimate of how the model will perform on new data?
Reveal answer
No. Repeatedly consulting the test set biases the evaluation, so the reported performance stops being an honest estimate.
A test set is meant to be consulted only once, at the end, to give an unbiased view of the finished model. Each time you check it and act on the result, the test set starts shaping your decisions, and the score you report no longer reflects unseen data.
What Hyperparameters Are
A hyperparameter is a setting chosen before training begins. The training process does not learn it from the data. Examples named in the source are the learning rate, the number of training epochs, and the size of each batch.
Because hyperparameters are fixed before training, you need a way to compare candidate values. The usual approach is to train with one candidate, measure how well the resulting model performs, and then compare that result with the other candidates. The data used for that measurement is what decides which settings you keep, so choosing it carefully matters.
Validation Data and Test Data Compared
A validation dataset is held out from training, just like a test set, and the two look alike. That resemblance is exactly why they are easy to confuse. The difference lies in how often each one is allowed to influence your decisions. A validation dataset is meant to be used many times while tuning. A test set should be used only once, at the end.
| Aspect | Validation dataset | Test set |
|---|---|---|
| When it is used | Repeatedly, during hyperparameter tuning | Once, at the end |
| What it informs | Which hyperparameter settings to keep | The final, unbiased estimate for the finished model |
| How often it is consulted | After each change to the settings | Only one time |
| Role in the process | Guides you through tuning | Final judge of the model |
Validation and test data look similar but serve different roles.
The validation set absorbs the repeated tuning, so it is the dataset that guides you. The test set stays in reserve as the final judge. If the test set has been used to choose settings, it no longer plays that role, and the evaluation it provides is compromised.
Following Data Through the Splits
Trace the three splits through a tuning process. The model learns from the training split. Each candidate setting produces a model that is scored on the validation split, and the best-scoring setting is kept. Only after that choice is made is the finished model judged on the test split, and that judgment happens once.
Tracing one tuning decision
You want to compare two learning rates, A and B. Both are trained on the training split. You need to decide which one to keep and then report a final score.
- 1Train with learning rate A
The model learns from the training split only. Its settings are fixed before training starts.
- 2Score A on validation
The validation split is used to measure how well A performs. This result is one input to the decision.
- 3Train with learning rate B and score on validation
The same procedure is repeated for the second candidate, again using the validation split for the measurement.
- 4Keep the better setting
Compare the two validation results and keep the setting that performed better. The test split has not been touched yet.
- 5Evaluate once on the test split
The finished model is scored on the test split a single time. This reported score is the unbiased view of the model.
The validation split chose the setting, and the test split produced the final reported score. The test split was consulted once.
Computing Accuracy One Chunk at a Time
Evaluating a model on a validation or test set requires computing its accuracy, which means counting how many predictions match the correct labels. With large datasets, this often cannot be done in one step, because memory limits usually mean only a small portion of the dataset can pass through the model at a time. The accuracy computation therefore works with chunks and combines the results.
The procedure iterates over a data loader. In each iteration the model receives one chunk, its predictions for that chunk are compared with the true labels, and the number of correct predictions is added to a running total. When the loop finishes, the running total is divided by the size of the whole dataset. Because each chunk has the same size as the batches used during training, the same procedure works for datasets of any size.
- The loop asks the data loader for one chunk at a time, so the whole dataset never has to sit in memory.
- Each chunk's correct count is added to the running total.
- Only after the loop finishes is the total divided by the size of the whole dataset.
Accuracy over a dataset of 1000 examples
A dataset has 1000 labeled examples. The data loader yields 10 chunks of 100 examples each, the same size as the training batch. The number of correct predictions in each chunk is 88, 91, 79, 95, 84, 90, 87, 92, 80, and 89.
- 1Start the running total at zero
Before any chunk is processed, no correct predictions have been counted.
- 2Add each chunk's correct count
After chunk 1 the total is 88. After chunk 2 it is 179. After chunk 3 it is 258, and the loop continues the same way through chunk 10.
- 3Reach the final total
Adding all ten counts gives 875 correct predictions in total.
- 4Divide by the whole dataset size
The accuracy is 875 divided by 1000, which is 0.875.
The accuracy over the full dataset is 0.875, computed without ever holding all 1000 examples in one pass.
The final accuracy depends only on the running total and the overall size of the dataset. That is why the procedure does not need the whole dataset resident in memory at once.
Mistakes That Bias Evaluation
Common mistake Using the test set to choose hyperparameters
What it looks likeTrying several learning rates, checking the test score after each, and reporting the best one.
Why it failsRepeatedly consulting the test set biases the evaluation, so the final reported performance stops being an honest estimate.
Do this insteadUse the validation set for every tuning decision and consult the test set once, after the settings are fixed.
Common mistake Treating validation and test data as interchangeable
What it looks likeAssuming that because a validation set and a test set both sit outside training, either can be consulted as often as needed.
Why it failsThe two look similar, but a validation set is meant to be used many times while tuning, whereas a test set should be used only once.
Do this insteadAssign each split its role: validation for tuning, test for the single final judgment.
Common mistake Assuming accuracy requires the whole dataset in one pass
What it looks likeTrying to push a very large validation set through the model in a single step.
Why it failsMemory limitations usually allow only a small portion of the dataset through the model at a time.
Do this insteadIterate over a data loader, accumulate the correct count for each chunk, and divide by the total dataset size at the end.
Common mistake Forgetting that a chunk size must match the training batch size when the procedure is set up around batches
Practice: Assigning Splits to Tasks
You are tuning a model. You have a training split, a validation split, and a test split. For each task below, decide which split should be used: (a) choosing between two batch sizes, (b) producing the number you report in your write-up, and (c) deciding whether to keep training for more epochs. Then explain what would go wrong if you used the test split for task (a).
Hints
- Ask which split is meant to be used many times while tuning.
- Ask which split should be consulted only once, at the end.
- Think about what happens to the reported score when the test set has already influenced a decision.
Key Takeaways
- Hyperparameters are settings chosen before training, such as the learning rate, the number of epochs, and the batch size, and a validation dataset is the held-out data used to find the best of them.
- A validation dataset is used many times during tuning, while a test set should be used only once, at the end, to give an unbiased view of the finished model.
- Repeatedly consulting the test set biases the evaluation, so the reported performance stops being an honest estimate.
- Accuracy on a large dataset can be computed chunk by chunk, adding each chunk's correct count to a running total and dividing by the size of the whole dataset at the end.