Searching Over Spaces That Are Not Grids

So far, a grid search has tuned the settings of one estimator. In this article the search is allowed to choose between two different estimators as well as their settings. The question it answers is: which combination of model and settings scores best? Before reading the code, it helps to see the problem the design solves. A grid is a set of values to try for each setting. When two model families need different settings, one shared set of keys cannot describe both of them.

The source uses two candidate families for this. The first family uses an SVC classifier, optionally with a scaling step in front of it. The second family uses a RandomForestClassifier, which needs no scaling. The SVC has gamma and C as its settings, while the RandomForestClassifier has max_features. Those parameters belong to different estimators, so the search space has to keep the two families separate.

Read the grid below once before you look at anything else. It is a list containing two dictionaries. The first dictionary covers the SVC family. The second covers the RandomForest family. Each dictionary's keys are the settings for that family only. The values in the code are illustrative choices made for this article, not values taken from the source, so focus on the structure rather than the specific numbers.

The search is run with the same pattern you already know from single-estimator grid search. The pipeline object is passed to GridSearchCV together with the list of grids, and the data is split first so that the test set stays untouched by the search. Notice that the two dictionaries do not share their keys. The SVC dictionary has gamma and C, and the RandomForest dictionary has max_features. The diagram shows how the list contains the two branches.

Read the diagram from the top. The list sits at the root, and each dictionary is one branch. Each dictionary's keys are the settings for that branch. Every branch is searched separately, and the search compares the best result from each branch against the others. The source explains the reason for this design directly: the two classifiers have different parameters to tune and need different preprocessing, so a list of search grids is the natural container.

The grid has to say which pipeline step each value belongs to. The rule is that the name of the step is used as the parameter name. That is why the key 'classifier' holds a list of estimator objects, and the key 'preprocessing' holds a list of preprocessing objects. The step name is the key itself, with no extra text.

Hyperparameters of an estimator are written differently. You write the step name, then two underscores, then the name of the hyperparameter. So 'classifier__gamma' means the gamma setting of whatever estimator sits in the classifier step, and 'classifier__C' means its C setting. The double underscore is the separator. It tells the search where the step name ends and the setting name begins.

Some estimators need a preprocessing step and some do not. When a step should be skipped, for example because a RandomForest does not need preprocessing, that step can be set to None. In the grid above, this is why the RandomForest dictionary lists preprocessing as [None] alone, while the SVC dictionary lists both StandardScaler() and None. The SVC family is therefore searched both with and without scaling, and the RandomForest family is searched without scaling only.

This is the practical reason the search is organized by dictionary. Different estimators need different preprocessing, and a single shared dictionary could not express that without trying combinations that make no sense. Putting a RandomForest with a scaler would be an invalid combination in this design, and separate dictionaries prevent it from being generated.

A single grid of values multiplies out: every combination of the listed values is one candidate. Each candidate is then cross-validated. The grid in this example uses cv=5, so every candidate is fitted five times. Count each dictionary separately, then add the totals. Do not count the preprocessing options once for the whole grid, because they belong only to the dictionaries that list them.

Each candidate setting is a complete pipeline configuration. For a candidate from the SVC family, the training data passes through the preprocessing step, which is either a scaler or a pass-through, and then into the classifier step, which is an SVC with that gamma and C. Grid search does this for every candidate in turn, scores each one with cross-validation, and keeps track of the best.

The path through the pipeline is the same for every candidate. Only the objects inside the two steps change. This is why one pipeline object can serve both model families: the grid decides what fills each step, and the pipeline decides the order in which the steps run.

After fit, the printed results report three things. best_params_ shows the winning combination, best_score_ shows its cross-validation score, and grid.score on the held-out test data gives the final check. When a multi-estimator search gives a result you did not expect, the most useful question is where the divergence began. The path has three checkpoints: the grid definition, the pipeline step names, and the printed attributes. Check them in that order.

Work through the checkpoints in order. If a key has a typo in its step prefix, the grid definition is the place the divergence began, even though the printed best_params_ looks like a plausible answer. If the grid is correct, check whether the step names in the pipeline match the prefixes you used. If both are correct, compare best_score_ with the test score: best_score_ comes from cross-validation during the search, and grid.score comes from held-out test data, so the two numbers answer different questions.