Grid search with GridSearchCV

Inside the search, each candidate value is tried on several training folds.

Programming

Why the Metric Is Part of the Search

Earlier, you learned to compute evaluation metrics from a model's predictions and the true labels. Used that way, a metric is a report written after a model is finished. In model selection, the metric plays a different role: it is the judge. When you choose between candidate models, or between candidate settings of one model, the number the search maximizes is the number that decides which configuration wins.

This article starts from a short, concrete question. Suppose you run a grid search over the same candidate values, with the same estimator and the same folds, but switch the metric from one scorer to another. Does the winner change? The answer is not automatic, and working through it shows exactly where the metric enters the procedure.

Scoring One Model with cross_val_score

Begin with a single model rather than a search. Consider a support vector classifier trained on the handwritten digits dataset, where the target has been turned into a yes or no question: is this digit a nine? You can ask cross_val_score to evaluate this same model with different scores.

The scoring argument controls the metric. If you leave it out, scikit-learn uses accuracy for classification. You can also pass the name of a metric as a string. That string is the entire interface: you do not import a metric function or write a wrapper around it.

python

from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

X, y = load_digits(return_X_y=True)
y_nine = (y == 9)
model = SVC(gamma=0.001)

acc_scores = cross_val_score(model, X, y_nine, cv=5)
auc_scores = cross_val_score(model, X, y_nine, cv=5, scoring='roc_auc')

print(acc_scores)
print(auc_scores)

from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

X, y = load_digits(return_X_y=True)
y_nine = (y == 9)
model = SVC(gamma=0.001)

acc_scores = cross_val_score(model, X, y_nine, cv=5)
auc_scores = cross_val_score(model, X, y_nine, cv=5, scoring='roc_auc')

print(acc_scores)
print(auc_scores)

Notes on the code

load_digits returns the feature matrix X and the digit labels y
y == 9 turns the digit labels into the yes or no target
With no scoring argument, accuracy is used
scoring='roc_auc' passes the metric as a string

Only the scoring string differs between the two calls. The model, the data, and the folds are the same, so any difference in the printed numbers comes from the metric alone. Each array holds one score per fold, which is why cross_val_score returns several values rather than one. Compare the two arrays to see how the metric alone changes the per-fold numbers.

Tracing a Grid Search Run, Candidate by Candidate

GridSearchCV applies the same idea across a list of candidate values. Suppose you give the search a deliberately weak grid of gamma values for the same SVC, ranging from very small to very large. Once a metric is named, the search evaluates each candidate by that metric and keeps the best one.

Inside the search, each candidate value is tried on several training folds. For each held-out fold, the scorer turns the predictions into a single number. Those numbers are averaged across folds, and the candidate with the highest average is kept. The scorer is therefore part of the selection loop itself, not a report written after the loop has finished. The string you pass controls the number every candidate is measured with.

What happens to each gamma value in turn: how is it trained and scored across folds, and how is the single best candidate kept?

Read the loop as a sequence of decisions. Each gamma value is trained and scored fold by fold, its fold scores are averaged, and the running best is updated only if the new average is higher. The loop ends when every candidate has been compared, and the surviving candidate is the one the search reports as best.

Switching the Scorer Changes the Winner

Now connect the two ideas. Keep the candidate gamma values, the SVC estimator, and the folds fixed. Run one search with the default accuracy scorer and a second search with the roc_auc scorer. Nothing about the candidates, the folds, or the estimator changes between the two searches. The search is the same procedure with a different yardstick.

For the same gamma candidates, how does the chosen best gamma differ when the scorer switches from accuracy to AUC?

The scorer enters at the scoring step and nowhere else. A different scoring string therefore changes which candidate sits at the top of the averages, and the rest of the procedure follows along. Whether the winner actually changes depends on how the two metrics rank the same candidates, which is exactly what you should predict before running the code.

python

from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

X, y = load_digits(return_X_y=True)
y_nine = (y == 9)
grid = {'gamma': [0.0001, 0.001, 0.01, 0.1, 1.0]}

acc_search = GridSearchCV(SVC(), grid, cv=5)
acc_search.fit(X, y_nine)

auc_search = GridSearchCV(SVC(), grid, scoring='roc_auc', cv=5)
auc_search.fit(X, y_nine)

print(acc_search.best_params_)
print(auc_search.best_params_)

from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

X, y = load_digits(return_X_y=True)
y_nine = (y == 9)
grid = {'gamma': [0.0001, 0.001, 0.01, 0.1, 1.0]}

acc_search = GridSearchCV(SVC(), grid, cv=5)
acc_search.fit(X, y_nine)

auc_search = GridSearchCV(SVC(), grid, scoring='roc_auc', cv=5)
auc_search.fit(X, y_nine)

print(acc_search.best_params_)
print(auc_search.best_params_)

Notes on the code

The grid lists the candidate gamma values to compare
The first search uses the default accuracy scorer
The second search differs only by scoring='roc_auc'
best_params_ reports the candidate the search kept

MEDIUM

Before running the code, write down which gamma you expect each search to pick. Then run both searches and compare best_params_. If the two picks match, explain which step of the loop would have had to produce the same ranking under both scorers. If they differ, name the scorer step as the place where the divergence begins.

Hints

Both searches share the same candidate list and folds.
Only the scorer differs, so only the ranking inside the averaging step can differ.

Your attempt (not saved)

Finding the Step Where a Surprise Begins

When two searches over the same grid disagree, the tempting move is to suspect the model or the data. A more productive habit is to trace the divergence to a specific place in the pipeline. The question to ask is which step first produced a different value, and whether that step is the one you expected to differ.

Which step (per-fold scores, fold means, or best-index selection) produces the final chosen parameter, and where would an unexpected winner first appear?

Read the pipeline from left to right. The per-fold scores come from the scorer, so a changed scoring string first shows up there. The fold means then average those scores per candidate, and the best-index selection keeps the candidate with the highest mean. An unexpected winner therefore first appears at the per-fold scores or the fold means, and the final selection step only reports what those earlier steps produced.

Scoring Strings for Each Task

The scoring argument accepts a family of named strings, and some are for classification while others are for regression. The strings are a convenient way to name a scorer, and each name maps onto a well-defined measure. Match the string to the task before you pass it, because a metric chosen for the wrong task will rank candidates on the wrong question.

Which scoring string (accuracy, roc_auc, neg_mean_squared_error, r2) belongs to which task type?

accuracy: classification. It is the default when no scoring argument is given.
roc_auc: classification, as used in the yes or no digit example, where the search ranks candidates by area under the ROC curve.
neg_mean_squared_error: regression. The name begins with neg because scikit-learn's convention for this scorer is that larger values are better, so the error is negated.
r2: regression, scoring how well the model's predictions explain the variation in the target.

Mistakes Beginners Make with Scorers

Common mistake
Assuming the default metric is always the right one
Why it fails
Classification defaults to accuracy when no scoring argument is given, so the search ranks candidates by accuracy even if you care about a different property.
Do this instead
State the scoring argument explicitly whenever the metric matters to your decision.
Common mistake
Expecting the same winner under any metric
Why it fails
The scorer is used inside the selection loop to rank every candidate, so changing it can change which parameter wins.
Do this instead
Predict the winner under each metric and verify it by comparing best_params_ from both searches.
Common mistake
Suspecting the model or data when two searches disagree
Why it fails
The candidate values, folds, and estimator are usually shared, so the divergence most often begins at the scoring step.
Do this instead
Trace the per-fold scores and fold means first, then the best-index selection.
Common mistake
Passing a regression scorer to a classification task
Why it fails
Each string maps onto a specific task, so a metric chosen for the wrong task ranks candidates on the wrong question.
Do this instead
Match the string to the task using the classification and regression groupings above.

Practising the Scorer Choice

EASY

A team is building a yes or no classifier for a binary medical screening task, and they want the search to reward the model that ranks positive cases above negative cases across all thresholds. Which scoring string would you pass to GridSearchCV, and why? Then explain which step of the search loop is where your choice changes the outcome.

Hints

The task is classification, and the goal is ranking across thresholds.
Think about which named string in the scorer table measures area under the ROC curve.

Your attempt (not saved)

MEDIUM

Suppose a colleague ran two grid searches over the same gamma candidates with different scoring strings, and the best gamma differs between them. Describe the order in which you would check the pipeline: per-fold scores, fold means, or best-index selection. State which step you would check first and what you would look for there.

Hints

The scorer produces the per-fold scores.
Work forward from the first step that could differ.

Your attempt (not saved)

Key Takeaways

The scoring argument names the metric that cross_val_score and GridSearchCV use, and a string such as 'roc_auc' is the entire mechanism.
The scorer ranks every candidate inside the selection loop: each candidate is scored per fold, the scores are averaged, and the highest average wins.
Changing only the scorer can change best_params_, so predict the winner under each metric and verify it by running both searches.
When two searches disagree, trace the divergence to the scoring step first, because the other inputs are usually shared.
Match each scoring string to its task: accuracy and roc_auc for classification, neg_mean_squared_error and r2 for regression.

Related concepts

More concepts starting with G

Why the Metric Is Part of the Search

Scoring One Model with cross_val_score

Tracing a Grid Search Run, Candidate by Candidate

Switching the Scorer Changes the Winner

Finding the Step Where a Surprise Begins

Scoring Strings for Each Task

Mistakes Beginners Make with Scorers

Practising the Scorer Choice

Key Takeaways

Related Topics

Related concepts