Accuracy and the Confusion Matrix

For classification, accuracy is the default.

Programming

Why the Metric Decides the Winner

Earlier evaluation work taught you to compute a metric from ground truth and a model's predictions. Model selection asks a different question: which setting of a model's parameters should you keep? To answer it, a search tool has to know which number to compare across candidate settings. That number is the scoring metric, and in scikit-learn you choose it with a single argument.

What do you think happens?

You run a grid search over gamma with the default scoring, which is accuracy for classification. Then you run the same search with scoring set to roc_auc, on the same training data. Do you expect both runs to keep the same gamma?

Yes, the grid and the data are the same, so the winner is the same
No, the metric is the comparison that decides the winner, so the winner can change

Reveal answer

Answer:

The winner can change. In the source's run, accuracy kept gamma=0.0001 and AUC kept gamma=0.01.

The fitting and the candidate settings are the same in both runs. Only the number used to compare candidates differs, and that is enough to change which setting wins.

This lesson is about that comparison number. You will trace where the scoring argument enters the search, read a pair of runs that disagree, and learn which scoring strings to reach for. The trace starts with the workflow itself.

Tracing Where Scoring Enters the Search

GridSearchCV tries each candidate value in a parameter grid, scores each one, and keeps the best. Scoring is the step that turns each candidate into a comparable number. The diagram follows the data from the parameter grid down to the best_params_ attribute, so you can see exactly where the metric is used.

Which step does the scoring string touch, and how does that choice flow through fold splits, candidate evaluation, and best_params_ selection? The scoring step sits at the end, where candidates are compared.

Notice the position of the scoring step. The metric does not affect fitting, and it does not affect which candidates are generated. It affects only the comparison at the end, and that comparison decides which candidate's settings you keep. This is why two searches over the same grid can return different winners.

Following One Candidate Through the Folds

The scoring step is applied per fold, not once to the whole training set. For a single candidate, the training data is split into folds. Each fold takes a turn as held-out data while the candidate is fitted on the remaining folds, and the chosen metric is computed on the held-out fold. The per-fold numbers are then averaged. That average is the single number grid search compares against the other candidates.

For each candidate, how do the folds get split, scored, and averaged to produce the single number that grid search compares? Here k stands for the number of folds, which the cv argument controls.

Because the same metric is used on every fold and then averaged, a different metric produces a different average for the same candidate. The folds and the fitted models stay the same. Only the per-fold numbers and their average change.

Reading the Accuracy and AUC Runs

The source uses a deliberately narrow grid that varies only gamma, so the effect of the metric is easy to see. The first search uses the default accuracy. The second uses AUC, which is requested with the string roc_auc. Both searches are fitted on the same training split and evaluated on the same test split.

Worked example

Two searches, two winners

The same gamma grid is searched twice on the same data. Once with the default accuracy, and once with roc_auc. Which gamma does each search keep, and why do they differ?

1
Identify the fixed parts
The grid, the classifier, the training split, and the test split are the same in both runs. Only the scoring argument changes.
2
Read the accuracy run
Under accuracy, the search keeps gamma=0.0001, and its cross-validation accuracy is 0.970.
3
Read the AUC run
Under AUC, the search keeps gamma=0.01, and its cross-validation AUC is 0.997.
4
Locate the cause
The fitting and candidate generation did not change. The scoring step at the comparison is the only place the two runs diverge, so it explains the different winners.
5
Check the held-out data
The source states that the cross-validation accuracy agrees with the test set accuracy in both cases, so the scores are not misleading for this run.

Result

Each search is correct by the metric it was given. Accuracy keeps gamma=0.0001, and AUC keeps gamma=0.01.

If I change scoring from accuracy to roc_auc, which candidate wins, and what changed in the scores that caused the switch? The grid is the same in both columns, and only the metric differs.

Scoring Strings for Classification and Regression

scikit-learn offers many scoring strings. The source highlights a small set that matters most. For classification, accuracy is the default. The string roc_auc asks for the area under the ROC curve, average_precision asks for the area under the precision-recall curve, and the f1 family covers the binary F1 score and its weighted variants. For regression, the commonly used strings are r2 for the R squared score, mean_squared_error for mean squared error, and mean_absolute_error for mean absolute error.

Scoring string	Task	Metric it asks for	Direction for the search
accuracy	Classification	Accuracy (the default)	Higher is better
roc_auc	Classification	Area under the ROC curve	Higher is better
average_precision	Classification	Area under the precision-recall curve	Higher is better
f1	Classification	Binary F1 score	Higher is better
f1_macro	Classification	F1 averaged across classes, unweighted	Higher is better
f1_micro	Classification	F1 from pooled counts across classes	Higher is better
f1_weighted	Classification	F1 averaged across classes, weighted by class size	Higher is better
r2	Regression	R squared score	Higher is better
neg_mean_squared_error	Regression	Mean squared error, negated	Higher is better after negation
neg_mean_absolute_error	Regression	Mean absolute error, negated	Higher is better after negation

Commonly used scikit-learn scoring strings. The f1 variant descriptions are standard definitions; check your installed version's documentation for exact behavior.

The source also shows how to list every available name by reading the scorer dictionary in the metrics module. That list is long and includes entries such as log_loss, adjusted_rand_score, and several precision and recall variants. The source's import path for that dictionary is an older one, so confirm it against your installed scikit-learn documentation. For everyday work, the names in the table are the ones to learn first.

Changing Only the Scoring Argument in Code

The same argument works in both tools. You can judge one model with cross_val_score and then search over settings with GridSearchCV using the same metric. The example below uses a narrow gamma-only grid in the spirit of the source. The grid values here are illustrative, and the source's own run is the one whose results are reported above.

python

from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.svm import SVC

param_grid = {"gamma": [0.0001, 0.01]}

# Judge one fixed model with the same metric the search will use
scores = cross_val_score(SVC(gamma=0.01), X_train, y_train, scoring="roc_auc", cv=5)
print(scores.mean())

# Default scoring: accuracy, same as omitting the argument
search_acc = GridSearchCV(SVC(), param_grid, scoring="accuracy", cv=5)
search_acc.fit(X_train, y_train)
print(search_acc.best_params_)

# Same grid, same data, different comparison number
search_auc = GridSearchCV(SVC(), param_grid, scoring="roc_auc", cv=5)
search_auc.fit(X_train, y_train)
print(search_auc.best_params_)

from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.svm import SVC

param_grid = {"gamma": [0.0001, 0.01]}

# Judge one fixed model with the same metric the search will use
scores = cross_val_score(SVC(gamma=0.01), X_train, y_train, scoring="roc_auc", cv=5)
print(scores.mean())

# Default scoring: accuracy, same as omitting the argument
search_acc = GridSearchCV(SVC(), param_grid, scoring="accuracy", cv=5)
search_acc.fit(X_train, y_train)
print(search_acc.best_params_)

# Same grid, same data, different comparison number
search_auc = GridSearchCV(SVC(), param_grid, scoring="roc_auc", cv=5)
search_auc.fit(X_train, y_train)
print(search_auc.best_params_)

Notes on the code

scoring="roc_auc" in cross_val_score returns one score per fold, using the same metric the search will compare
scoring="accuracy" is the default for classification, so passing it explicitly does not change the results
Only the scoring argument differs between the two GridSearchCV runs

Best practice

Choose the scoring string before you run the search, and write it into the code explicitly. An implicit default is easy to forget when you later compare two searches, and an explicit argument makes the comparison number visible in the code.

Mistakes When a Selection Looks Surprising

Common mistake
Assuming the search optimizes the same thing as your final evaluation
What it looks like
You report a model using AUC but select its parameters with the default accuracy, then wonder why the chosen gamma looks wrong for AUC.
Why it fails
The metric enters at the comparison step. If the comparison uses accuracy, the winner is the best by accuracy, whatever you report later.
Do this instead
Pass the same scoring string to GridSearchCV that you care about, and to cross_val_score when you judge a single model.
Common mistake
Looking for the bug in the fitting code when the winner changes
What it looks like
Two searches return different best_params_, and you rerun the fit repeatedly to find the cause.
Why it fails
Fitting and candidate generation do not depend on the metric. The difference lives in the scoring comparison.
Do this instead
Check the scoring argument first, then compare the candidate scores under that same metric.
Common mistake
Treating an error metric as if larger were better
What it looks like
You pass mean_squared_error to a search and expect the search to prefer the largest error values.
Why it fails
A search keeps the highest score, so a loss that should be minimized has to be negated to fit that rule.
Do this instead
Use the negated scoring name, such as neg_mean_squared_error, and confirm it in your installed version's documentation.
Common mistake
Assuming that because the cross-validation score is high, the choice is correct for every metric

Predicting Which Metric Changes the Winner

MEDIUM

A colleague runs a gamma grid search with the default scoring and keeps gamma=0.0001. They then rerun it with roc_auc on the same data and get gamma=0.01. Before you check anything, predict which step of the workflow could be responsible, then state what you would print to verify your explanation.

Hints

Look at where the scoring step sits in the pipeline: it affects the comparison, not the fitting.
Compare the cross-validation scores under the same metric that each search used.
Check the scoring argument in each GridSearchCV call before anything else.

Your attempt (not saved)

Key Takeaways

The scoring argument, passed as a string, sets the metric that cross_val_score and GridSearchCV use to evaluate candidates.
Scoring enters a grid search at the comparison step: fitting and candidate generation do not depend on the metric, so changing it can change best_params_.
When a selected parameter surprises you, check the scoring argument first, then compare candidate scores using that same metric.
accuracy is the default for classification, and common alternatives include roc_auc, average_precision, and the f1 family; common regression strings include r2 and the negated error metrics.

Related concepts

More concepts starting with A