Understanding Grid Search

A classifier such as an SVC has settings that you choose before training begins. Two of them, gamma and C, strongly affect how well the model performs on data it has not seen. Choosing good values for these settings is called parameter tuning. This article looks at one of the simplest ways to do it, called grid search, and at the small program that implements it with nothing more than nested for loops.

A simple grid search works in three moves. First, you define a set of candidate values for each parameter. Second, you try every combination of those values. Third, for each combination you train and evaluate a classifier and keep the combination that produces the best score. The search never looks at a combination that was not on the grid, and it never adjusts the grid while it runs.

The idea is easy to state, but it is easy to lose track of what happens during the search. The source notes that the program makes a single pass over the combinations and prints only the winner. It does not print a table of every score. This means that the count of models trained, and the order in which they were trained, are not shown in the output. You have to reason about them from the code structure, which is the skill this article builds.

The outer loop assigns each gamma value in turn. For every gamma, the inner loop walks through every C value. The inner body therefore runs once for each pair, and every pair is complete: each model uses one gamma and one C together. The source writes the loop headers as a grid of two lists, one for gamma and one for C, which is why the search is described as a grid.

It helps to picture the parameter grid as a tree. The root represents the whole search. Each gamma value is a branch leading off the root. Under each gamma branch, every C value becomes a leaf. Each leaf is one complete pair, and each leaf corresponds to one trained model. Counting the leaves gives the total number of models.

Every pass through the inner body does the same three things, and each has a specific job. The SVC constructor takes the current gamma and C, so the classifier is rebuilt for every pair rather than reused from an earlier pair. The fit call trains that classifier on the training data. The score call measures how well the trained classifier does on the test data and returns one number. Then the program compares that number with the best score seen so far. Three steps, one combination.

The search carries its memory in two variables. The variable best_score begins at 0. After each evaluation, the program compares the new score with best_score using a strict greater-than test. Only when the new score is larger does the program overwrite best_score and build the best_parameters dictionary, which records the C and gamma values that produced it. Every other combination leaves both variables untouched.

The output contains four lines. The first two come from the data split performed before the loops start. The training set has 112 samples and the test set has 38. Those two numbers add up to 150, the number of samples in the iris dataset, which shows that the split is a complete partition of the data with no sample left out. The last two lines come from the search: the best score, printed with two decimal places, and the best parameters, printed as a dictionary with C and gamma as keys.

The printed winner is the final state of best_score and best_parameters after the last combination has been tried. In this run the winning C is the largest value in the grid and the winning gamma is the smallest. The output reports that fact without explaining it. Whether the best setting sits at an edge of the grid cannot be judged from this output, because the scores for the other combinations are never printed.