Grid search for hyperparameter tuning

Grid search is a way of trying out different settings and keeping the combination that works best. Before you can search over anything, though, you need an object to search over. In scikit-learn that object is often a Pipeline, which bundles several processing steps into one unit. This article starts with the pipeline itself, because grid search later changes the steps inside it.

Read the line that builds pipe as a recipe rather than as a computation. It creates a pipeline with two steps. Nothing has been learned from data yet. The rest of this article follows that pipe object through three stages: how it is defined, how its steps are named and ordered, and how a grid changes those steps before a search evaluates them.

A Pipeline is built from a list of named steps, and the steps run in the order given. Defining the pipeline records those steps, but it does not fit anything yet. No scaling is computed and no classifier is trained at the moment the object is created. The pipeline is a single object that stores which steps exist and in what order.

The diagram shows the run order. Each named step receives the output of the step before it. The name is what identifies each step, so the same pipeline can later be referred to from outside by those labels.

The names matter as much as the estimators. Each name is a label for its step, and you use that label to refer to the step from outside the pipeline. In the pipeline above, 'preprocessing' and 'classifier' are the two labels. The order of the list is the order of the steps, so the preprocessing step comes before the classifier.

The label is also how a grid reaches inside the pipeline. When you want to change a setting on a particular step, you write a key that starts with that step's name. The key below is split into its three parts so you can see which part names the step and which part names the estimator's parameter.

A common goal is to compare two different classifiers, such as RandomForestClassifier and SVC. Those two classifiers have different parameters to tune, and they may need different preprocessing. One fixed dictionary of parameters cannot describe both cases cleanly. So the search uses a list of search grids, and each grid in the list covers one combination of choices.

To put an estimator into a step, you use the step's name as the parameter name in the grid, and the value is the list of estimators to try for that step. The grid below has two entries. The first tries SVC with scaling, and the second tries a random forest with no scaling.

Each dictionary in the list defines its own set of candidates, and the candidates are evaluated separately. The tree below shows how the grid expands into concrete candidate pipelines. Notice that the two branches never share parameters: the SVC branch has C values, and the random forest branch has tree-count values.

Sometimes one candidate needs a step that another candidate does not. The random forest in the example does not need preprocessing. To skip a step for a particular candidate, you set that step to None. In the grid, the second entry sets the preprocessing step to None, so the search tries the random forest without a scaling step, while the first entry still scales the data before the SVC.

Skipping is a per-candidate choice. Setting a step to None does not remove the step from the pipeline definition for every candidate. It only skips that step for the candidate in which the value appears. Other candidates in the search can still include the step.

When a grid search does not behave as you expected, the cause is usually one of three things: a step's name, the value you assigned to a step, or the structure of the list. Narrow the problem down in that order. Start with the names, because the name is how the grid finds the step it is changing. Only after the names match should you check the values, and then the list structure.