Simple Grid Search

A grid search is a method for finding good parameter values for a classifier. You list candidate values, try them, and keep the combination that scores best. Once you learn that a grid search can run in parallel, a natural belief follows: if one core makes the search slow, giving the search every core on the machine should make it as fast as possible. That belief is partly correct, but it is incomplete in a way that can cost time and memory. This article starts with a complete script, traces what it does, and then builds the reasoning needed to use parallelism well.

The article has three goals. First, you will follow a simple grid search step by step and count the models it trains. Second, you will see why grid search splits so neatly across cores, and where that splitting runs into limits. Third, you will examine why the reported best score can mislead you when the parameters are chosen using the test data.

A simple grid search is a set of nested for loops that trains and evaluates a classifier for every combination of candidate parameter values. The source introduces it using two parameters of a support vector classifier, SVC: gamma and C. The outer loop chooses a gamma value and holds it fixed. The inner loop then cycles through every C value. When the inner loop finishes, the outer loop advances to the next gamma, and the inner loop starts again from the first C value. Because of this structure, every pair of values is visited exactly once.

Inside the inner loop, four steps repeat for every combination. A new SVC is built with the current gamma and C. The model is fitted on the training data. The fitted model is scored on the test data. Finally, an if statement compares that score with best_score. If the new score is higher, both best_score and the best_parameters dictionary are overwritten. Every combination is trained and scored, but the stored result is replaced only when a new score beats the current best.

The stored answer starts empty. best_score begins at a value below any real score, and best_parameters is an empty dictionary. Only the if comparison can change either variable. The comparison replaces them only when a new score is higher than the current best_score. A lower score is evaluated and then ignored. The print statements run once, after every loop has finished, so they report the final stored state rather than any intermediate one.

Read the diagram as a trace. Only three evaluations change the stored answer. The scores 0.85 and 0.93 are computed and then discarded, because each is lower than the best seen so far. This is the core rule of the script: the search keeps every score in its head only long enough to compare it, and it keeps just the winning combination.

Grid search and cross-validation are embarrassingly parallel. The term means that the work divides into pieces that never need to talk to each other. Consider one combination of parameter settings evaluated on one cross-validation split. Building that model needs only the training portion of that split and the chosen parameter values. It does not need the result of any other combination or any other split. Each piece can therefore run on its own, in any order, on any core.

When a grid search script fails or gives a strange result, the cause usually sits in one of a few places. Trace the script in order and check each place before blaming the model.

The source section on this implementation is titled to flag a danger. In the script, every combination is scored on the test set, and the combination with the highest test score is reported as best. The parameters were therefore picked to fit the test data. The reported best score is no longer an unbiased estimate of how the model will perform on new data. The selection process has leaked information from the test set into model choice.