Cross-Validation Score

Before you can judge a model, you have to know which version of it to judge. A grid search answers that question by trying several hyperparameter settings and keeping the one that scores best. In this article you will start from a finished search and work backward from its printed output. The search is fitted on the training portion of a breast cancer dataset, and then three numbers are reported. Each of those numbers comes from a different place, and knowing where each one comes from is the core skill this article builds.

The setup assumes that a pipeline object and a parameter grid already exist, and that the cancer dataset has been loaded. What matters for evaluation is the flow of data: the search is fitted on the training split, and the test split is kept back until the last step. Before reading further, decide for yourself which of the reported scores should be higher, and why. Think about which score comes from data the search used while choosing its settings, and which comes from data it never touched.

A GridSearchCV run reports three things: the best parameters, the best cross-validation score, and a score on held-out data. The best cross-validation score is the attribute best_score_. It reports the average score of the winning combination across the folds that were used to select it. This is the number the search itself optimized, so it describes how the chosen settings performed on the data used to pick them.

The held-out score comes from a different call. The call grid.score(X_test, y_test) uses the test split, which was never passed to fit. No part of the search adjusted itself to that data. This is why the test-set score is the number that speaks to how the model is expected to behave on new data. In the run described by the source, the two values are close: 0.99 from cross-validation and 0.98 on the test set.

The diagram shows the direction of the flow. The training split feeds the search, the folds produce both the best settings and the cross-validation score, and the test split enters only at the very end. Keeping that direction in mind helps you locate the source of any surprising number.

GridSearchCV performs cross-validation for every combination in the grid. For each combination, the training data is split into five folds. The model is trained on four of them and scored on the fifth, and the scores are averaged. This happens for every combination, and the combination with the highest average is the winner. The steps below follow that sequence in order, from one candidate setting to the final test score.

The best parameters are a dictionary, and the source describes its contents for one run. The chosen classifier is an SVC with kernel='rbf', and the chosen preprocessing step is StandardScaler. The two remaining entries are the hyperparameters the grid varied, C=10 and gamma=0.01. Each of these values is the outcome of the averaged fold scores described above, not of the test split.

The keys in best_params_ use a naming pattern that you will see often when a pipeline is involved. Two keys, classifier and preprocessing, hold the objects chosen for the pipeline steps. The other two keys, classifier__C and classifier__gamma, are the hyperparameters that the grid varied. A pipeline addresses a parameter inside one of its named steps by joining the step name and the parameter name with a double underscore.

Suppose a colleague reports that a search gave a test-set score far below its cross-validation score. The useful question is not whether the search is broken, but at which step the numbers stopped agreeing. The checklist below follows the data flow in order, so each check narrows the problem to one step.

A grid search stores one row of results for each parameter setting it tried. Each row records the score from every cross-validation split, along with the mean and standard deviation across those splits. When the search covers two parameters, the natural picture is a heat map. One axis holds one parameter, the other axis holds the second parameter, and each cell is colored by its mean validation score. To draw it, you pull out the mean scores and reshape them into a rectangle whose rows and columns follow the two parameters.

The reshape step only works when the number of stored scores equals the number of cells in the grid. A 6 by 6 grid has 36 cells, so the flat list must contain exactly 36 values. Here is a small generated example with made-up numbers, not real model scores, that shows how a flat list becomes rows.

The schematic above is generated for illustration. Each cell stands for one parameter setting, and its color stands for accuracy. Light cells mark high accuracy and dark cells mark low accuracy. Reading the picture means asking where the light cells are, where the dark cells are, and how those regions line up with the two axes. In the diagram, the best cell sits in the interior of the grid rather than on an edge.

The source uses an SVC model to show how dramatic a heat map can be when parameters matter. In that plot, many settings of C and gamma produce accuracy around 40 percent, which is poor, while other settings reach around 96 percent. Moving between settings changes the result from very bad to very good. The message is that the two parameters strongly control performance.

The same plot supports two checks you should make on any heat map. First, do the ranges you chose actually change the outcome? Second, does the best region sit inside the grid, or does it touch an edge? The source notes that the optimum values in its plot were not on the edges, which suggests the search ranges were wide enough to capture the peak.

A uniform heat map is the hardest picture to read, and the first reaction is often to throw the run away and start over with a different model. That reaction is usually premature. A uniform picture is a clue, and it points to a small set of specific causes that you can check one at a time. The source describes three heat maps produced from deliberately misspecified grids, each with a different color arrangement, and each arrangement points to a different diagnosis.

The two causes look different on the page. When the range is too narrow, the whole map tends to look similar, so the search has not explored a region where the score changes. When one parameter has little effect, the color varies along the other axis but stays constant along the axis holding the unimportant parameter. Naming the pattern is the first step toward deciding what to change. The source's comparison also reminds you that a uniform heat map is a prompt to investigate, not proof that the model has failed.

Adjusting the parameter grid after looking at cross-validation results is a legitimate and useful habit. The heat map guides those adjustments: it shows where accuracy changes, where it does not, and where the best region presses against the edge of the search. The adjustments should be made using cross-validation scores on the training data, which is where the heat map comes from. The test split stays untouched until the final score is computed.