Train-Test Splitting and Overfitting
Grid search with cross-validation tries each parameter combination in a grid and evaluates each one by cross-validation on the training data.
The Single Number That Looks Like an Answer
Suppose you have tuned a support vector machine (SVM) and printed a single number that looks like the model's accuracy. It is tempting to treat that number as the final answer about how the model will behave on new data. This article examines that belief. The belief is partly right, and the part that is wrong is exactly where grid search with cross-validation produces most of its confusion.
What do you think happens?
A grid search has produced two numbers: one from the cross-validation run that selected the parameters, and one from scoring the fitted search object on a held-out test set. Do those two numbers come from the same data?
Reveal answer
No. The cross-validation number comes from validation folds carved out of the training data, while the test score comes from a held-out test set that the search never used to pick parameters.
The two numbers answer different questions. The first measures how the selected setting performed during the search. The second is the generalization estimate, and it is only trustworthy because the test set stayed out of the loop.
Tracing One Pass Through the Search
Grid search with cross-validation is a loop. It takes one parameter setting, evaluates that setting with cross-validation on the training data, and compares the result with the best result seen so far. Then it moves to the next setting. Follow the loop below, which shows one setting being evaluated and the search repeating over the rest of the grid.
- 1Take the first parameter setting from the grid.
- 2Run cross-validation for that setting: split the training data into several parts, train on some parts, validate on the remaining part, and rotate which part is held out.
- 3Record the mean validation accuracy and compare it with the best result so far.
- 4Repeat for every other setting in the grid.
- 5Keep the setting with the highest mean validation accuracy and use it to fit a fresh model on the whole training set.
Where the Test Set Sits in the Workflow
Cross-validation on its own is a way to evaluate one algorithm on one dataset. Grid search on its own is a way to try many parameter settings. In practice the two are almost always combined, and people often say cross-validation when they mean grid search with cross-validation. The combined method tries each parameter setting and evaluates each one with cross-validation, so the choice of parameters rests on validation performance rather than on a single lucky split.
The data is split once into a training set and a test set before the search begins. Cross-validation replaces the single training and validation split used earlier, but a held-out test set is still needed to judge the final choice. The diagram traces which data feeds the search and which data waits until the end.
The Parameter Grid as a Dictionary
A parameter grid is a plain Python dictionary. Each key is the name of a hyperparameter, written exactly as the model's constructor expects it. Each value is a list of candidate settings to try. The grid describes the search space before any training happens.
- Keys are hyperparameter names spelled as the SVC constructor expects them.
- Values are plain Python lists of candidate settings.
- Six values each produce 36 combinations in total.
Counting the Models in a Grid Search
The source states that the number of models trained grows with the grid size times the number of folds. Working through the grid above makes that rule concrete.
Models trained for the SVM grid
The grid has six values for C and six values for gamma, and cross-validation uses five folds. How many models are trained during the search, and how many more are trained after it?
- 1Count the combinations
Six values of C combined with six values of gamma gives 36 parameter settings.
- 2Multiply by the folds
Each setting is evaluated with five cross-validation folds, so 36 times 5 gives 180 models trained during the search.
- 3Add the refit
After the search, the winning setting is used to fit one more model on the whole training set.
The search trains 180 models during cross-validation and one final model on the whole training set. The test set is not used in any of these steps.
Running GridSearchCV
scikit-learn packages the whole procedure into the GridSearchCV class. You give it the estimator to tune, the parameter grid, and the cross-validation strategy. The object then behaves like a classifier. The search runs when you call fit, and afterwards the object can predict and score immediately, because the best setting has already been refit on the whole training set.
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.svm import SVC
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
param_grid = {
"C": [0.1, 1, 10, 100, 1000, 10000],
"gamma": [0.0001, 0.001, 0.01, 0.1, 1, 10],
}
grid = GridSearchCV(SVC(), param_grid, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)
print(grid.best_score_)
print(grid.score(X_test, y_test))- 1The data is split once into training and test sets before any search begins.
- 2The parameter grid is a dictionary of hyperparameter names mapped to candidate lists.
- 3GridSearchCV receives the estimator, the grid, and a five-fold cross-validation strategy.
- 4fit runs the full search on the training data only.
- 5best_params_ and best_score_ report the winning setting and its mean validation accuracy.
- 6score on the held-out test set gives the generalization estimate, and the test set was never used during the search.
Reading best_params_, best_score_, and best_estimator_
After fit, the grid search object exposes several attributes. Three of them matter most. Each one stores something different, and each one comes from a different stage of the procedure.
| Attribute | What it stores | Where its value comes from |
|---|---|---|
| best_params_ | The hyperparameter setting that scored highest in the search | The comparison of mean cross-validation results across the grid |
| best_score_ | The mean cross-validation accuracy of that winning setting on the training set | The validation folds carved out of the training data |
| best_estimator_ | The model refit with the best setting on the whole training set | A fresh fit on all of the training data, using the winning parameters |
What the three main attributes of a fitted GridSearchCV object measure
Why the CV Score and Test Score Differ
best_score_ is the mean cross-validation accuracy on the training set. The score on the held-out test set is the generalization estimate. The two are different measurements, even though both are accuracies. One was produced by data the search used to choose parameters. The other was produced by data the search never touched.
| Measurement | Data that produced it | Role in the workflow |
|---|---|---|
| best_score_ | Validation folds inside the training set | Reports how well the winning setting did during the search |
| Test score | Held-out test set that was never used to select parameters | Estimates how well the selected parameters generalize |
The cross-validation score and the final test score come from different data
Where the Workflow Goes Wrong
The test set does not appear anywhere in the search loop. That absence is the reason the test score is a fair check afterward. The diagram compares a correct workflow with one in which test-set results feed back into the choice of hyperparameters.
Common mistake Tuning the grid after looking at the test score
What it looks likeThe search prints a low test score, so the grid is widened and the search is run again until the test score improves.
Why it failsThe test set has now influenced which parameters were chosen. The final test number is no longer a fair estimate of generalization.
Do this insteadAdjust the grid using cross-validation results on the training data only. Score the test set once, at the end.
Common mistake Treating best_score_ as the final answer about new data
What it looks likeReporting only best_score_ as the model's accuracy in a project summary.
Why it failsbest_score_ is a mean validation accuracy on training data, not the held-out generalization estimate.
Do this insteadReport the test score as the generalization estimate, and state that best_score_ measures the search.
Common mistake Assuming cross-validation replaces the test set
What it looks likeSplitting no data off at all and relying only on the folds from GridSearchCV.
Why it failsThe source states that cross-validation replaces the earlier training and validation split, but a held-out test set is still needed to judge the final choice.
Do this insteadSplit off a test set once before the search begins.
Common mistake Assuming that equal values for best_score_ and the test score mean the two always match
Practice Predicting the Workflow
A colleague's grid search prints best_score_ equal to 0.91 and a test score of 0.84. Explain which data produced each number, and say which one you would report as the expected accuracy on new data. Then describe one change to the workflow that would keep the test score fair.
Hints
- Ask which data was used while the parameters were being chosen.
- The test set should appear only once, after the final choice is made.
- Look at whether the grid was adjusted after any test result was seen.
Key Takeaways
- A parameter grid is a dictionary whose keys are hyperparameter names and whose values are lists of candidate settings, and the number of models trained grows with the grid size times the number of folds.
- GridSearchCV runs cross-validation on the training data for every setting, keeps the best one, and refits it on the whole training set so predict and score are available right away.
- best_score_ is the mean cross-validation accuracy on the training set, while the score on the held-out test set is the generalization estimate; they are different measurements.
- The test set must stay out of the search loop. If test results influence the choice of hyperparameters, the test score stops being a fair check.