GridSearchCV and Hyperparameter Search
GridSearchCV can search over models and preprocessing steps at once, because the parameter grid is a list of dictionaries that each describe one family of candidates.
Searching Beyond One Model's Settings
When you build a classifier, you usually have to choose settings that the model does not learn from the data by itself. These settings are hyperparameters. The usual manual approach is to tune one model's settings, run a few experiments, and compare the results by hand. That approach answers one question: which hyperparameter values work best for this one model?
A more complete question is which combination of model, preprocessing step, and hyperparameter values works best. GridSearchCV is the tool that asks this question automatically. It does not care whether the thing it is searching over is a number, a model object, or a preprocessing step. It tries each candidate in a grid, evaluates it, and reports which candidate scored best. The source's running example compares a Support Vector Classifier (SVC) against a RandomForestClassifier on the cancer dataset, and it also asks whether the data should be scaled with StandardScaler or left unchanged.
Reading the Parameter Grid
The central object in this workflow is the parameter grid, and it is a Python list. Each entry in the list is a dictionary, and each dictionary describes one family of candidates. A family is a set of models or options that belong together. Each dictionary maps a key to a list of values to try. The key 'classifier' holds the model objects themselves, and the key 'preprocessing' holds the preprocessing choices. Keys such as 'classifier__gamma' and 'classifier__C' point at hyperparameters of a model.
- The list holds two independent dictionaries, one for each model family
- Entry one covers the SVC branch with its gamma and C values
- Entry two covers the RandomForestClassifier branch with its
max_featuresvalues - Both entries offer the same two preprocessing options:
StandardScaler()or no scaling (None)
The grid above is an illustrative version written for this article. Its structure follows the source's pattern: a list of two dictionaries, one for the SVC branch and one for the RandomForestClassifier branch. The specific values are chosen only to make counting practical.
Read the diagram from the top. The list contains two families that are independent of each other. The search explores the candidates inside each family, but it does not combine the RandomForestClassifier with the SVC's gamma and C values. Each entry defines only the keys that apply to its own model. That is why the SVC's C and gamma keys appear only in entry one, and the max_features key appears only in entry two.
Reading the Double Underscore Keys
The double underscore is how a pipeline refers to a setting inside one of its steps. The part before the double underscore names the step, and the part after it names the parameter of that step. So 'classifier__C' means the C parameter of whatever object is stored under the 'classifier' step. The same rule gives 'classifier__gamma' and 'classifier__max_features'.
The 'preprocessing' key works differently. It is not a parameter of a model at all. It names a step, and the values listed under it are the possible objects for that step, written directly. In the grid above those objects are StandardScaler() and None, meaning the data is either scaled or passed through unchanged.
Counting Candidates Before You Run
Before running a grid, it helps to know how much work it will do. The rule has two steps. First, count the candidates inside each entry: the number of candidates in one entry is the product of the lengths of its value lists. Second, add the entries together to get the total. Each candidate is one combination of values from one entry.
What do you think happens?
How many candidate combinations does the illustrative grid evaluate in total?
Reveal answer
38
Entry one has 2 preprocessing options, 1 classifier object, 4 gamma values, and 4 C values, giving 2 times 1 times 4 times 4 = 32 candidates. Entry two has 2 preprocessing options, 1 classifier object, and 3 max_features values, giving 2 times 1 times 3 = 6 candidates. Adding the entries gives 32 + 6 = 38.
Counting the candidates and the fits
Count the candidates in the illustrative grid, then count the model fits when GridSearchCV uses cv=5.
- 1Count entry one
Multiply the value-list lengths: 2 preprocessing options times 4 gamma values times 4 C values gives $2 \times 4 \times 4 = 32$ candidates.
- 2Count entry two
Multiply the value-list lengths: 2 preprocessing options times 3
max_featuresvalues gives $2 \times 3 = 6$ candidates. - 3Add the entries
Add the two families to get $32 + 6 = 38$ candidates in total.
- 4Count the fits
Each candidate is evaluated once per cross-validation fold, so cv=5 multiplies the work: $38 \times 5 = 190$ fits.
The grid evaluates 38 candidates and runs 190 fits in total.
Running the Search on a Pipeline
Once the grid is defined, the search is run as usual. First the data is split into training and test sets with train_test_split, using random_state=0 so the split is repeatable. Then GridSearchCV is built around a pipeline object, which the source refers to as pipe, and it is asked to use five cross-validation folds. Calling fit runs the whole search on the training data only. The test set is held back until the end.
- The pipeline's step names 'preprocessing' and 'classifier' must match the keys in the grid
train_test_splitwithrandom_state=0 holds back a repeatable test set- GridSearchCV with cv=5 runs the search on the training data only
- best_params_, best_score_, and score on the test set report three different things
The snippet uses pipe but does not define it. The source assumes a pipeline named pipe already exists in your session, with a step named 'preprocessing' and a step named 'classifier'. The pipeline definition in the code above is included so that the example runs on its own. If pipe is missing, Python raises a NameError on the GridSearchCV line, before any search happens. This is the first place to look when the code fails. X and y are also assumed to hold your feature and target data.
Inside a pipeline, each candidate is evaluated by passing the training data through the chosen preprocessing option and then into the chosen classifier. Changing one step's option changes that path, while the rest of the pipeline stays the same.
Read the diagram as a single candidate's journey. Only one option from each step is active for any given candidate. Swapping the preprocessing choice from StandardScaler() to None changes what the classifier receives, and swapping the classifier object changes which model produces the score. Because the search covers both steps together, it can find that scaling helps one model and not another.
Reading the Printed Output
The printed output needs careful reading. The best_params_ attribute reports the winning combination across both models and the preprocessing step, not just the best hyperparameter value. The best_params_ result is a dictionary that includes the winning classifier object, the winning preprocessing object, and the winning hyperparameter values. In the source's cancer dataset run, the selected classifier is an SVC, the selected preprocessing is a StandardScaler, and the selected values are stored under classifier__C and classifier__gamma. Reading this dictionary tells you which branch of the grid won, not only which number was best.
{'classifier': SVC(C=..., gamma=...), 'preprocessing': StandardScaler(), 'classifier__C': ..., 'classifier__gamma': ...}\n<best cross-validation score>\n<score on the test set>The output above is a shape rather than a set of numbers. The actual values depend on the data and on the grid you run, so your printed values will differ from one run to the next.
Two Scores, Two Questions
Two scores appear in the output, and they answer different questions. The best_score_ attribute is the best cross-validation score from the training data. It is the score the search used to choose the winner, so it is somewhat optimistic about the chosen model. The score method, called on the held-out test set, reports how the chosen model performs on data the search never saw. That second number is the final performance check.
| Score | Data it uses | When it is produced | What it tells you |
|---|---|---|---|
| best_score_ | Training data, through cross-validation folds | During the search, when fit runs | How well the winning candidate did during selection; somewhat optimistic |
| score on test set | Held-out test data the search never saw | After the search, when called explicitly | How the chosen model performs on new data; the final check |
The two scores answer different questions.
Mistakes When Writing Grids
Common mistake Writing a single dictionary that mixes model-specific keys
What it looks likeA single dictionary holds both classifier__gamma and classifier__max_features
Why it failsEach entry defines only the keys that apply to its own model, so a mixed dictionary combines settings that do not belong together.
Do this insteadPut each model's keys in its own dictionary in the list.
Common mistake Misspelling a step name in the grid
What it looks likeUsing 'classfier__C' when the pipeline step is named 'classifier'
Why it failsThe grid's key names must match the pipeline's step names exactly, so the search cannot find the step.
Do this insteadCheck that each key before the double underscore matches a step name in the pipeline.
Common mistake Running the search without defining the pipeline
What it looks likeCalling GridSearchCV(pipe, ...) in a session where pipe was never created
Why it failsPython raises a
NameErroron the GridSearchCV line, before any search happens.Do this insteadDefine the pipeline first, with step names that match the grid.
Common mistake Treating best_score_ as the final performance
What it looks likeReporting only best_score_ as how well the model will perform
Why it failsbest_score_ was used to choose the winner, so it is somewhat optimistic.
Do this insteadReport the score on the held-out test set as the final performance check.
Common mistake Assuming the grid's count is the number of values rather than the number of combinations
Practice Counting and Reading a Grid
A grid has two entries. Entry A has a preprocessing key with 2 options and a classifier__C key with 3 values. Entry B has a preprocessing key with 2 options and a classifier__max_depth key with 5 values. First, how many candidates does the grid evaluate? Then, with cv=4, how many fits does the search run?
Hints
- Count entry A first by multiplying the length of each value list.
- Count entry B the same way, then add the two entries.
- Multiply the total number of candidates by the number of cross-validation folds to get the fits.
Key Takeaways
- GridSearchCV searches over models and preprocessing steps at once, because the parameter grid is a list of dictionaries and each dictionary describes one family of candidates.
- Within one dictionary, the candidate count is the product of the value-list lengths; across dictionaries, the counts add. Multiplying by the number of cross-validation folds gives the number of fits.
- A key such as classifier__C uses the double underscore to point at a parameter inside a named pipeline step, while a key such as preprocessing lists the step's possible objects directly.
- best_params_ reports the winning combination across models and preprocessing, best_score_ is the cross-validation score used to choose it, and the score on the held-out test set is the final performance check.