Tuning Model Parameters with Grid Search
The accuracy measured on the dataset used for parameter tuning is not a trustworthy estimate of how the model will perform on new data.
Why a High Score Can Mislead
Imagine you have built a model and tuned its parameters until it reaches a high accuracy score on your data. The natural next step feels like the finish line: report the number and declare the model good. This article looks at why that final step can quietly go wrong, even when every calculation was done correctly.
The trap is not a mistake in arithmetic. The score is an accurate count of how the model did on one particular set of data. The error lies in treating that count as a forecast for data the model has never seen. Before you can see why, you need to understand what parameter tuning does to the data it is measured on.
What Parameter Tuning Actually Does
Parameter tuning is a search. You try a number of different parameter settings, measure how each one performs, and keep the setting that scores best. Each candidate setting is a configuration the model could be built with, and the search compares those configurations against one another using a measurement of performance.
The important detail is what the measurement is taken on. When the measurement uses the test data, that test data stops being a neutral judge. It has helped decide which configuration survived. Every candidate that was tried and rejected was rejected partly because of how it scored on that same data, and the winner was chosen for the same reason.
Tracing a Tuning Run
Follow one tuning run as the search moves through its candidates. Each pass tries another setting against the same tuning data. The search keeps whichever setting has scored best so far, so the tuning score it reports can only go up or stay the same from one trial to the next. The diagram below traces this progression.
Notice what the loop does not do. It never consults fresh data. Each trial is judged by the same examples, so the search is steadily fitting its choice to those examples. By the time the loop ends, the chosen setting has been picked because it matched the quirks of the tuning data, not only because it reflects a general pattern.
Why Tuning Data Stops Being Neutral
Overfitting in parameter tuning means a model can perform well on the tuning dataset but fail to generalize to new data. The high score is partly a product of selection rather than a sign of general skill.
A common misconception is that the accuracy measured on the tuning dataset is the model's true performance. The reasoning sounds plausible: the model was tested on data, it scored well, so it must be good. The source flags this belief as overly optimistic, and in many cases simply wrong, because of how the parameters were chosen.
A configuration that happened to fit the quirks of the tuning data was picked because of those quirks. Nothing guarantees that those quirks will appear in new data. A setting that looked best only because it matched one dataset's accidental patterns will look worse, sometimes much worse, once it meets examples it was never selected against.
| Question | Tuning dataset | Independent dataset |
|---|---|---|
| Was it used to choose parameters? | Yes | No |
| Was it used to create the model? | Yes, in part of the search | No |
| Is its score a fair estimate of new-data performance? | Not on its own | Yes |
| What role does it play? | Guides the search | Final judge |
Accuracy measured on the tuning dataset compared with accuracy on an independent dataset
A Tuning Run Reported Too Early
Reporting a score from the wrong dataset
A team tries a range of parameter settings on one dataset, keeps the setting with the best score on that dataset, and then reports that score as the expected performance on future data. Trace why the reported number cannot be trusted. This scenario is illustrative and uses no measured figures.
- 1Identify the data used for tuning
The dataset that scored each candidate setting is the tuning dataset. It guided which setting survived.
- 2Notice the selection effect
The winning setting was chosen partly because it fit that dataset well. Its high score is partly a product of that choice.
- 3Check whether new data was involved
No independent dataset was used to create the model or tune it, so nothing tested the setting on examples it was not selected against.
- 4Judge the reported number
The score accurately describes the tuning dataset, but it is not a fair estimate of performance on new data.
- 5Repair the process
Hold back an independent dataset that was never used for creating or tuning the model, and evaluate the final setting on it.
The reported score is an accurate count on the tuning data but an unreliable forecast for new data. An independent dataset is needed to produce a fair estimate.
Keeping Evaluation Data Separate
The fix follows from the same logic that motivated splitting data into training and test sets at the start. Each dataset needs a single, clear role. Data that shapes the model or its parameters belongs to the tuning side. Data kept aside for the final judgement must stay out of both creating the model and tuning it.
In the pipeline, the tuning data feeds the parameter search, and the search produces a chosen model. The independent data never feeds the search. It enters only at the end, where the chosen model is measured on it. Only that kind of dataset gives a fair estimate of performance.
Mistakes That Undermine a Tuning Result
Common mistake Reporting the tuning-set score as the model's true performance
What it looks likeA model scores highly on the data used to pick its parameters, and that number goes straight into a report as expected accuracy.
Why it failsThe tuning score reflects selection as well as skill. The chosen setting was picked partly because it suited that data.
Do this insteadReport performance only from an independent dataset that played no part in creating or tuning the model.
Common mistake Using the final judging data to adjust parameters
What it looks likeAfter a poor first result, the same evaluation data is used again to try more settings until the score improves.
Why it failsOnce the data has helped choose the configuration, it can no longer serve as a neutral judge.
Do this insteadKeep the evaluation data untouched during tuning and use it only once the final model is fixed.
Common mistake Assuming a high score on one dataset means the model will generalize
Common mistake Treating more tuning trials as always better without a separate check
Checking Your Understanding of Selection Bias
What do you think happens?
After grid search picks the setting with the highest score on the tuning data, can that same tuning data also be used to report the model's final performance?
Reveal answer
No, because it helped choose the setting
Once a dataset has been involved in choosing the model or its parameters, it cannot also serve as the final judge. A fair estimate needs an independent dataset that was not used to create or tune the model.
A colleague tunes a model by trying twenty parameter settings, each scored on the same evaluation set, and then reports the best score as the model's expected accuracy on new customers. Explain in your own words why this number may be too optimistic, and describe the change to the process that would give a fair estimate.
Hints
- Think about what happened to the evaluation set during the search.
- Ask whether any of the data used to judge the final model was involved in choosing it.
- Consider which dataset would be untouched by the tuning process.
Key Takeaways
- Accuracy measured on the dataset used to tune parameters is not a trustworthy estimate of how the model will perform on new data.
- Parameter tuning is a search that compares candidate settings on data, so the chosen setting can be selected partly because it fits that data.
- Repeated tuning against the same data can produce overfitting in parameter tuning, where the model performs well on that data but fails to generalize.
- A fair evaluation requires an independent dataset that was not used to create the model or to tune its parameters.