Grid Search

Once you can measure how well a model generalizes to unseen data, the next question is how to make it generalize better. One lever is the model's parameters. Many learning algorithms expose settings that change how the model behaves, and the values you choose for the important ones can decide whether the model performs well or poorly on new data. This article follows one tuning method, grid search, and works through what it does, how its parameter grid is written, and how to find the cause when a search does not behave as expected.

Finding good parameter values is difficult. You need to know what each parameter controls before adjusting it, and you usually cannot guess the best setting in advance. Still, this kind of search is required for almost every model and dataset, which is why it is treated as a routine task with standard tools. The most widely used of these standard approaches is grid search.

The name comes from the way the search is organized. You pick a set of candidate values for each parameter. Those candidate sets are then combined so that every value of one parameter is paired with every value of the others. The resulting collection of pairings forms a grid, where each cell is one complete parameter setting to be tested. Grid search does not try to be clever about which cells to skip. It visits all of them.

Because the method tests every combination, its cost grows quickly. Each added parameter multiplies the number of settings, so a grid with several parameters and several candidate values per parameter can contain many cells. The method is most useful when you have a small number of parameters that you already believe matter.

Consider a kernel SVM with an RBF (radial basis function) kernel, the form implemented in the SVC class. This model has two important parameters: the kernel bandwidth, called gamma, and the regularization parameter, called C. Both control how the model fits the training data, so both are candidates for tuning.

The same counting can be done in Python. The itertools.product function produces every pairing of two lists, which is exactly the set of cells in the grid. The sketch below uses two short lists so the output is easy to read. It illustrates the idea and is not a full tuning procedure.

Once a model sits inside a pipeline, its parameters are reached through the pipeline, so the grid has to say which step each parameter belongs to. A pipeline is a sequence of named steps, and each step can hold an estimator that has its own parameters. The rule is to write the step name, then two underscore characters, then the parameter name, all inside one string.

Read the key 'svm__C' from left to right. The part before the double underscore names the step, which is svm. The part after it names the parameter, which is C. The double underscore is the boundary between the two. It is a required separator, and a single underscore would not be read the same way.

The grid is an ordinary Python dictionary. Each key is a pipeline-qualified parameter name, and each value is the list of candidate values the search should try for that parameter. The example below builds a small pipeline with a scaler as its first step and an SVC as its second step, which is named svm. The pipe object and the training and test splits are assumed to exist already in the session.

The call to GridSearchCV looks the same as it would for a single estimator. The pipeline is passed in as the first argument, and the grid is passed as the parameter grid. The only difference lies in how the grid's keys are written. Reusing the same search pattern with a pipeline therefore needs no new function or special class.

A plain estimator exposes parameters such as C directly. Inside a pipeline the same parameter is reached through the step that holds it. The before and after versions below show the same parameter written two ways. Only the key changes, so the difference is entirely in how the search routes the value.

When a grid search does not behave as expected, the most useful first check is the keys of the grid. A key that does not match a step name followed by a double underscore and a real parameter name will not reach the estimator it was meant for. The flowchart below turns that check into a sequence of questions you can follow in order.