Reading cv_results_ from a Grid Search
For that you need the full results table, which is stored in the cv_results_ attribute.
A Search That Repeats Itself
A grid search tries candidate settings of a model's parameters and scores each one. Most of the time, GridSearchCV tries every combination in the grid you give it. That sounds thorough, but it can quietly waste effort. Some parameters only matter when another parameter has a particular value, and a grid that ignores this relationship will evaluate candidates that are really the same model over and over. In this article you will see how that happens, how to write the parameter grid so it avoids it, and how to read the results table well enough to notice when a result does not match what you expected.
What do you think happens?
How many candidates does the search evaluate, and how many are genuinely different models?
Reveal answer
Four candidates are evaluated, but only two are genuinely different models.
The grid multiplies every value of each parameter together, so 1 kernel times 2 values of C times 2 values of gamma gives 4 candidates. With kernel set to linear, gamma is not used by the model, so the two candidates that differ only in gamma are repeats of the same linear model with C set to 1 and the same with C set to 10.
Conditional Parameters and Wasted Candidates
A conditional parameter is one that only has an effect when another parameter takes a particular value. The clearest example is SVC, where the kernel parameter decides which other parameters are used. With kernel set to linear, the model is linear and only C is used. With kernel set to rbf, both C and gamma are used, and parameters such as degree are not. The kernel acts as a switch, and the parameters attached to other kernels are simply inactive under the linear setting.
This matters when you build a grid. A single ordinary grid that lists kernel, C, and gamma together combines every kernel with every value of C and every value of gamma. For the linear kernel, gamma is not used at all. So each linear candidate that carries a different gamma value is just a repeat of the same model. Trying different values of gamma under the linear kernel is a waste of time, and the larger the grid, the more of this repetition you pay for.
Counting the candidates in one combined grid
A single grid has kernel ['linear'], C [1, 10], and gamma [0.1, 1]. How many candidates are evaluated, and how many distinct linear models does that produce?
- 1Count the values in each parameter
There is one kernel value, two C values, and two gamma values.
- 2Multiply the counts
The grid evaluates every combination, so the count is $1 \times 2 \times 2 = 4$ candidates.
- 3Check which parameters the model uses
With kernel set to linear, only C is used. Gamma has no effect on the model.
- 4Count the distinct models
Each C value gives one linear model, so there are 2 distinct models. The other 2 candidates repeat them.
The grid evaluates 4 candidates, but only 2 of them are different models. Half of the effort goes to repeats.
Splitting the Grid into Dictionaries
GridSearchCV accepts param_grid as a list of dictionaries. Each dictionary in the list is expanded into an independent grid, and the grids do not share their combinations. The search builds the candidates for the first dictionary, builds the candidates for the second dictionary, and evaluates all of them. Each dictionary can fix a parameter by giving it a list with a single entry. In the first dictionary below, kernel is always rbf, and both C and gamma still vary. In the second dictionary, kernel is always linear and only C varies. One param_grid can therefore describe two different shapes of search.
- The list holds two dictionaries, so the search builds two independent grids.
- A single-entry list such as ['rbf'] fixes kernel for that grid.
- The estimator is an SVC, and the list is passed in the same position an ordinary dictionary would take.
- cv=5 sets five cross-validation splits for every candidate.
The values in this example are illustrative. The first dictionary produces 3 times 3 = 9 candidates, and the second produces 3 candidates, for 12 in total. Notice that the value 0.1 for C appears in both dictionaries, so it is evaluated twice: once under rbf and once under linear. That is expected. Each group contributes its own rows to the results, and the two groups never mix their values into one another's combinations.
Reading the cv_results_ Table
After fitting, the best_params_ attribute names one winning candidate, and best_score_ gives its mean cross-validation score. Neither value tells you how the other candidates compared. For that you need the full results table, which is stored in the cv_results_ attribute. The table records each candidate, so you can check which parameters varied for each row and where a result differs from what you expected. The code below converts the results into a DataFrame and transposes it. The transposition is a display choice that makes the candidates easier to scan on a page. It is not part of the search itself.
- best_params_ names the single winning candidate.
- best_score_ is the mean cross-validation score of that winner.
- The
DataFrameholds the whole table, and .T transposes it so each candidate reads as a column.
To read the table, look at the parameter columns first. A column that changes from row to row is a parameter that this candidate varied. A column whose value stays fixed across a group of rows is a parameter that the grid held constant for that group. For the linear candidates in the example above, the kernel value is always linear and the C column changes, while the gamma column has no value for those rows. Those empty entries show up as NaN.
Tracing an Unexpected Result
When a search result does not match what you expected, the cause usually sits in one of a few places. The same symptom can have different causes, so it helps to work through the possibilities in a fixed order. Start with the grid, then the table, then the validation setup.
- 1Find the row whose score surprised you, and read its parameter columns to see exactly which values produced it.
- 2Check which dictionary in
param_gridgenerated that row by looking at the fixed parameter, such as the kernel value. - 3Confirm whether the parameter you expected to matter is actually used under that row's setting, since a conditional parameter may be inactive.
- 4Look for NaN entries that explain a blank value in a column, and check whether the candidate was supposed to have a value there.
- 5Check whether tied candidates share the top score, because a tie can make the winner that best_params_ reports look more decisive than it is.
- 6Confirm the cv setting, because a result from one splitting strategy cannot be compared directly with a result from another.
Choosing the Cross-Validation Strategy
The cv parameter controls how each candidate is validated, and GridSearchCV handles it the same way cross_val_score does. Choosing a strategy means deciding how the training data is divided into training and validation portions, and how the scores for each candidate are produced from those divisions. The next two subsections explain the default behaviour and the single-split alternative.
Mistakes Beginners Make with Grids and Results
Common mistake Writing one combined dictionary when a parameter only matters under one value of another
What it looks likeA single dictionary with kernel, C, and gamma that generates repeated linear models for each gamma value.
Why it failsThe grid evaluates every combination, so candidates that differ only in an unused parameter are repeats that cost time.
Do this insteadUse a list of dictionaries, with one dictionary per shape of search, such as one for rbf with C and gamma and one for linear with C only.
Common mistake Assuming the list of dictionaries multiplies the grids together
Why it failsEach dictionary is its own independent grid, and the candidates are combined as a union, so the total is the sum of the grid sizes.
Do this insteadCount each dictionary separately and add the totals.
Common mistake Treating a NaN entry as a bug in the search
What it looks likeA gamma column with NaN values for every linear row.
Why it failsThe candidate has no gamma setting because the model does not use one, so the blank is correct.
Do this insteadRead the NaN as a missing parameter that was never applicable to that candidate.
Common mistake Trusting best_params_ as the full story
Why it failsIt names one winner and its mean score but says nothing about how the other candidates compared.
Do this insteadRead the cv_results_ table to see the runner-up candidates and any ties.
Common mistake Comparing scores produced under different cv strategies
Why it failsA single split and five folds produce scores from different divisions of the data, so they are not directly comparable.
Do this insteadKeep the cv setting fixed when comparing candidates in one search.
Practice Reading a Results Table
You fit a search with two dictionaries. The first fixes kernel to rbf and varies C and gamma. The second fixes kernel to linear and varies only C. Your results table shows a row where kernel is linear, C is 1, and gamma is NaN. Explain why the NaN is expected, and state how many rows in total you should expect if the first dictionary has 2 C values and 3 gamma values and the second has 2 C values.
Hints
- Ask which dictionary could have produced a row with kernel set to linear.
- Count the first dictionary as the product of its C and gamma counts, then add the second dictionary's count.
- The gamma NaN means the candidate was never given a gamma value.
Key Takeaways
- A full cross-product of parameters can waste effort when some parameters only matter under certain conditions, such as gamma under a linear SVC kernel.
- Passing
param_gridas a list of dictionaries creates independent grids whose candidates are combined by union, so each dictionary contributes its own rows and nothing is multiplied across dictionaries. - The cv_results_ table lists each candidate, so the parameter columns show which values varied, where NaN marks an inapplicable parameter, and where ties or unexpected scores appear.
- GridSearchCV defaults to stratified k-fold for classification and k-fold for regression, and any splitter can be passed as cv, including a single train/validation split from ShuffleSplit.