Grid Search with Cross-Validation

Suppose you have written a grid search for an SVC model and it has finished running. The search does not hand you a single answer. It tried many parameter settings, scored each one with cross-validation, and kept a record of everything it did. That record is where the real information lives. It shows how the model's performance changes as the parameters change, and it is the place to go when you want to know whether the search was informative at all.

This article follows the order of operations inside a grid search. As you read, keep asking three questions: which values are being chosen right now, which data is being used, and where is the best result being saved? Most mistakes in grid search code come from losing track of one of those three things. We will trace the enumeration of combinations, the single mean score that cross-validation produces for each one, and the structured record the search leaves behind.

Grid search tries every combination of the candidate hyperparameter values. You write down a list of candidate values for each hyperparameter, and the search visits each pairing. The total number of combinations is the product of the lengths of the lists. Two lists of length 2 give 2 times 2, or 4 combinations. Three gamma values and two C values give 6.

A grid search represents this enumeration by placing one loop inside another. The outer loop picks one value of the first hyperparameter. The inner loop then runs through every value of the second hyperparameter for that outer value. When the inner loop finishes, the outer loop moves to its next value and the inner loop starts over from the beginning of its list. The important consequence is that the outer loop completes its first value before the second outer value begins.

The diagram shows the traversal as a sequence. Each step either advances the inner loop to its next C value or advances the outer loop to its next gamma value, which resets the inner loop. The search never needs to build the full list of combinations in advance. It simply runs through them in order.

Once a combination of gamma and C is chosen, the search does not judge the model from a single train and validation split. Cross-validation splits the data into several folds, trains and evaluates the model on different subsets, and records a score for each split. The source code calls cross_val_score with cv=5, which asks for five splits. Each fold is used for validation exactly once.

Here $s_i$ is the score from split $i$. The source code computes this average with np.mean(scores), and that single number is what the search compares against the best score it has seen so far. This is why a grid search with cross-validation produces one number per combination, even though each combination was evaluated several times underneath. The average prevents the final number from reflecting one lucky or unlucky split.

After a grid search has been fitted, its results are stored in an attribute called cv_results_. The source describes it as a dictionary that stores all aspects of the search. A dictionary of this kind maps names to lists, with one entry per quantity recorded. It holds a great deal of detail, so it is rarely read directly. The usual approach is to convert it into a pandas DataFrame, which gives you a table you can display, filter, sort, and plot.

The DataFrame is the same information as the dictionary, arranged so that every parameter setting becomes a row and every recorded quantity becomes a column. Row i holds the parameter values for one setting together with that setting's split scores, mean, and standard deviation. Each row is therefore a summary of every split used to score that setting, not a single cross-validation run. The diagram shows four illustrative rows, each pairing a parameter setting with its position in the table.

Each row contains the individual split scores along with a mean and a standard deviation. Before you trust the summary values, check that they agree with the splits they summarize. The mean should equal the average of the five split scores. The standard deviation should measure the spread of those same five scores around that average. If either disagrees, something in the chain from parameters to scores has gone wrong.

The source gives one real row for context: with C = 0.001 and gamma = 0.001, the mean test score is 0.366 and the standard deviation is 0.011. The individual split values for that row are not reproduced here, so the example below uses its own illustrative numbers to show the check.

Here $s_i$ is a split score and $\bar{s}$ is the mean of the five splits. A useful habit is to read a row this way: first look at the individual split scores, then confirm that the mean is their average, and only then accept the spread as meaningful.

The source notes that the search covered a two-dimensional grid, varying C and gamma together. A table with one row per combination is accurate, but when two parameters change at once it is hard to see patterns across both axes. For a two-dimensional grid, the source recommends a heat map.

The procedure is to extract the mean validation scores first, then reshape them so that one axis corresponds to C and the other to gamma. Once arranged that way, each cell of the picture is one parameter combination, and the colors show how the score moves as you travel along either parameter. A flat list of scores does not show which pair produced which result, so the reshaping step matters.

The heat map also gives a sanity check on the loop. If the map does not have one cell for each combination you intended to try, the grid lists or the reshaping step are wrong. A two-by-two grid should produce exactly four cells. Missing or duplicated cells point back to the loop structure from earlier in the article.

When a grid search fails or gives a suspicious answer, the failure usually sits at one of a few specific points. Learning to locate them is more useful than memorizing the code. The items below are worth checking every time you read a search loop.