Understanding Model Parameters
Grid search is a method for tuning model parameters that tries every combination of the parameter values of interest.
Why Parameter Values Change Model Performance
Once you can measure how well a model generalizes to unseen data, the next question is how to make it generalize better. One lever is the model's parameters. Many learning algorithms expose settings that change how the model behaves, and the values you choose for the important ones can decide whether the model performs well or poorly on new data.
Finding good values is hard. You need to know what each parameter controls before you adjust it, and you usually cannot guess the best setting in advance. This search is needed for almost every model and dataset, so it is treated as a routine task with standard tools. The most widely used of those standard approaches is grid search. In this article you will trace how a grid is built from candidate values, how to count the settings it contains, and how the search visits them.
Tracing a Parameter Dictionary
Before any search runs, each parameter you want to tune needs a set of candidate values. A common way to hold this in Python is a dictionary, where each parameter name is a key and its list of candidate values is the value stored under that key. The dictionary does not yet describe any single model. It only describes which options are allowed for each parameter.
Read the diagram from left to right. Each parameter name owns a list of candidates. A single trial takes one value from each list and combines them into one complete setting. The lists themselves are not settings; they are options. This distinction matters for everything that follows.
Building the Grid from Candidate Lists
Grid search is a method for tuning model parameters that tries every combination of the parameter values of interest.
The name comes from how the search is organized. You pick a set of candidate values for each parameter. The candidate sets are then combined so that every value of one parameter is paired with every value of the others. The resulting collection of pairings forms a grid, and each cell of that grid is one complete parameter setting to be tested. Grid search does not try to be clever about which cells to skip. It visits all of them.
A small example makes the layout concrete. Suppose one parameter has three candidate values and another has four. Put the three values down the rows and the four values across the columns. Every cell in the resulting table pairs one row value with one column value, so the grid has 12 cells, one for each setting that would be tested.
Counting the Combinations
What do you think happens?
If one parameter has 3 candidate values and another has 4, how many cells does the grid contain?
Reveal answer
12
Every value of the first parameter is paired with every value of the second, so the count is 3 times 4, which is 12. Adding the counts gives 7, which counts only the options and not the pairings.
The counting rule follows directly from the pairing. Because each value of one parameter pairs with each value of the others, the number of settings is the product of the number of candidates for each parameter. Each added parameter multiplies the total.
- 1N is the total number of cells, which is the number of parameter settings grid search will test.
- 2n_1 through n_k are the numbers of candidate values for each of the k parameters.
- 3The total is the product of those counts, so every candidate for one parameter is combined with every candidate for the others.
Counting the Grid for a Kernel SVM
A kernel SVM with an RBF kernel has two important parameters, C and gamma. You want to test the values 0.001, 0.01, 0.1, 1, 10, and 100 for C, and the same six values for gamma. How many combinations does grid search try, and how are they laid out?
- 1Count the candidates for C
C has six candidate values: 0.001, 0.01, 0.1, 1, 10, and 100.
- 2Count the candidates for gamma
gamma also has six candidate values, the same six listed above.
- 3Multiply the counts
Every C value pairs with every gamma value, so the total is $6 \times 6 = 36$.
- 4Lay out the grid
Place gamma down the rows and C across the columns. The table then has six rows and six columns, and each of its 36 cells is one parameter setting.
Grid search tries 36 combinations, one for each cell of a 6 by 6 table of settings.
Visiting Every Cell in Turn
After the grid is built, the search has to visit each cell. The Python itertools.product function produces every pairing of two lists, which is exactly the set of cells in the grid. The sketch below uses it on the two candidate lists from the worked example. It is an illustration of the idea, not a full tuning procedure.
36
[(0.001, 0.001), (0.001, 0.01), (0.001, 0.1)]- product pairs every item of the first list with every item of the second.
len(grid)counts the cells, which matches 6 times 6.- grid[:3] shows the first three settings in the order product produces them.
The diagram shows one possible visiting order, the one that product generates for these lists. The first list varies more slowly and the second varies more quickly, so gamma changes at every step while C stays fixed until all six gamma values are tried. The essential point does not depend on the order: every cell is visited, and no cell is skipped.
How Cost Grows with Each Parameter
Because the method tests every combination, its cost grows quickly. Each added parameter multiplies the number of settings, so a grid with several parameters and several candidate values per parameter can contain many cells. The table below keeps six candidates per parameter and shows how the count grows as parameters are added.
| Number of parameters | Candidates per parameter | Combinations tried |
|---|---|---|
| 1 | 6 | 6 |
| 2 | 6 | 36 |
| 3 | 6 | 216 |
Growth of a grid with six candidates per parameter (generated from the multiplication rule)
Mistakes in Reading and Building a Grid
Common mistake Treating a single set of values as the whole grid
What it looks likeWriting C = 1 and gamma = 0.1 once and calling that the grid.
Why it failsThose two values form one setting, which is one cell. A grid needs a candidate list for each parameter so that many settings can be formed.
Do this insteadGive each parameter a list of candidate values, such as six values for C and six for gamma, and let the pairings form the cells.
Common mistake Adding the candidate counts instead of multiplying them
What it looks likeReasoning that six values for C and six for gamma give 12 settings.
Why it failsEach C value is paired with each gamma value, so the pairings multiply. Adding counts only counts the options, not the combinations.
Do this insteadMultiply the counts: $6 \times 6 = 36$ combinations.
Common mistake Assuming the cost stays the same when a parameter is added
What it looks likeExpecting a third parameter with six candidates to add only a few trials to a 36-cell grid.
Why it failsThe count multiplies with each parameter, so the total grows to $6 \times 6 \times 6 = 216$.
Do this insteadRecount the grid whenever a parameter is added and decide whether the extra trials are worth their cost.
Common mistake Skipping cells that look unlikely to matter
What it looks likeDropping gamma values that seem too large because they feel unpromising.
Why it failsGrid search does not try to be clever about which cells to skip. Removing cells by hand changes the method into a different, partial search.
Do this insteadKeep every combination in the grid, or change the candidate lists themselves before the search starts.
Common mistake Choosing parameters to tune without knowing what each one controls
Practice Counting a Three-Parameter Grid
A grid search has three parameters. The first has 4 candidate values, the second has 5 candidate values, and the third has 2 candidate values. How many combinations does grid search try, and how would you explain the count to someone who had added the three numbers together?
Hints
- Start with the number of candidates for each parameter, then multiply all three numbers together.
- Adding 4, 5, and 2 gives 11, but that counts the options rather than the settings.
- Work it in two steps: first 4 times 5, then the result times 2.
Key Takeaways
- Grid search tunes model parameters by trying every combination of the candidate values you choose, so no cell of the grid is skipped.
- Each parameter has a list of candidate values, and a single trial takes one value from each list to form one complete setting.
- The number of combinations is the product of the candidate counts, so the grid for a kernel SVM with six values each for C and gamma has $6 \times 6 = 36$ cells.
- Each added parameter multiplies the total, so grid search is best kept to a small number of parameters that you already believe matter.