Cross-Validation and Model Selection

The scoring argument, passed as a string, sets the metric that cross_val_score and GridSearchCV use to evaluate candidates.

Programming

Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

Set 10 test points aside; they take no part in any choice.
Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

Why Model Selection Needs a Number

Earlier evaluation work taught you to compute a metric from ground truth and a model's predictions. Model selection asks a different question: which setting of a model's parameters should you keep? To answer it, a search tool has to know which number to compare across candidate settings. That number is the scoring metric, and in scikit-learn you choose it with a single argument.

The most useful way to learn this idea is to run the same classifier under two different scores and look at what comes back. The rest of this article follows that experiment, then shows you where the scoring choice sits inside the search so you can reason about any surprising result.

What the Scoring Argument Controls

Definition

The scoring argument, passed as a string, sets the metric that cross_val_score and GridSearchCV use to evaluate candidates.

The string 'roc_auc' asks for the area under the ROC curve instead of accuracy. If you leave scoring out, the result is the same as saying accuracy, because accuracy is the default for classification. The same argument works in both tools. That means you can judge a single model with cross_val_score, then search over settings with GridSearchCV using the same metric, so the single-model judgment and the search agree about what 'good' means.

Tracing One Candidate Through the Loop

GridSearchCV tries each candidate value in a parameter grid, scores each one, and keeps the best. Scoring is the step that turns each candidate into a comparable number. The diagram follows one candidate through that loop, so you can see where the metric enters and what happens to each candidate's scores before a winner is chosen.

Notice the position of the scoring step. The metric does not affect fitting, and it does not affect the candidates themselves. It affects only the comparison at the end, and that comparison decides which candidate's settings you keep. This is why two searches over the same grid can return different winners: the candidates were fitted identically, and only the ranking rule changed.

How Folds Cover the Training Data

Each candidate's score is an average over cross-validation folds, so it helps to see how the folds divide the data. The diagram below is a generic illustration of four folds over four slices of training data, not a figure from a specific dataset. In each fold, one slice is held out for validation and the remaining slices are used for training. Across the folds, each slice is held out exactly once, so every row contributes to one validation score.

Read the diagram as a rule: the slice that is the validation set in one fold is training data in every other fold. The fold scores are then averaged into the single number that the search compares across candidates.

A Narrow Grid Scored Two Ways

The source uses a deliberately narrow grid, with only gamma values, so the effect of the metric is easy to see. The first search uses the default accuracy. The second uses AUC. Both are fitted on the same training split and evaluated on the same test split. Only the scoring argument differs between the two runs.

Worked example

Reading the trace of two searches over one grid

The same gamma grid is searched twice on identical splits. One search ranks candidates by accuracy and the other by AUC. Which setting does each search keep, and what score supports each choice?

1
Identify what is held constant
The grid, the training split, and the test split are the same in both runs. Fitting is therefore identical, and the candidates are identical.
2
Identify what changes
Only the scoring argument changes. The comparison at the end ranks the same candidates by a different number.
3
Read the accuracy search
Under accuracy, the search keeps gamma=0.0001, and its cross-validation accuracy is 0.970.
4
Read the AUC search
Under AUC, the search keeps gamma=0.01, and its cross-validation AUC is 0.997.
5
Check the agreement with the test set
The source states that the cross-validation accuracy agrees with the test set accuracy in both cases, so the accuracy-based winner is not a cross-validation artifact in this run.

Result

Each search is correct by the metric it was given. The two searches disagree on the winner because the metric changed, not because one search is buggy.

What do you think happens?

If you rank the same gamma candidates by AUC instead of accuracy, which gamma value does grid search keep?

gamma = 0.0001
gamma = 0.01

Reveal answer

Answer:

gamma = 0.01

Under AUC, the search kept gamma=0.01 with a cross-validation AUC of 0.997. The source also notes that this setting was better in accuracy in this run, so the metric change did not simply reverse the ranking here. The lesson is that the metric determines the winner for this grid and data, and AUC is not always superior.

The source is explicit that the finding is not that AUC is always superior. The finding is that the choice of metric changed which setting was selected for this grid and this data. A different grid or dataset could give a different winner under either metric.

python

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {"gamma": [0.0001, 0.01]}
search = GridSearchCV(SVC(), param_grid, scoring="roc_auc", cv=5)
search.fit(X_train, y_train)
print(search.best_params_)

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

param_grid = {"gamma": [0.0001, 0.01]}
search = GridSearchCV(SVC(), param_grid, scoring="roc_auc", cv=5)
search.fit(X_train, y_train)
print(search.best_params_)

Notes on the code

param_grid holds only the candidate gamma values to compare
scoring='roc_auc' is the only line that changes the ranking rule; remove it and the default accuracy is used
best_params_ reports the setting kept after the comparison step

python

from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

scores = cross_val_score(SVC(gamma=0.01), X_train, y_train, cv=5, scoring="accuracy")
print(scores.mean())

from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

scores = cross_val_score(SVC(gamma=0.01), X_train, y_train, cv=5, scoring="accuracy")
print(scores.mean())

Notes on the code

cross_val_score takes the same scoring string as GridSearchCV
Use it to judge one model before searching over settings with the same metric

Scoring Strings for Each Task

scikit-learn offers many scoring strings. The source highlights a small set that matters most. The table lists them by task type, and these are the names worth learning first.

Task type	Scoring string	What it measures
Classification	accuracy	Default for classification
Classification	roc_auc	Area under the ROC curve
Classification	average_precision	Area under the precision-recall curve
Classification	f1, f1_macro, f1_micro, f1_weighted	Binary F1 score and its weighted variants
Regression	r2	R squared score
Regression	mean_squared_error	Mean squared error
Regression	mean_absolute_error	Mean absolute error

The most important scikit-learn scoring strings, grouped by task type

The source shows how to list every available name by reading the scorer dictionary in the metrics module. That list is long and includes entries such as log_loss, adjusted_rand_score, and several precision and recall variants. The source's import path for that dictionary is an older one, so confirm the path in the documentation for your version. The table above is the set to learn first.

Mistakes When a Selection Looks Wrong

Common mistake
Assuming the search is broken when the winner is surprising
What it looks like
A grid search keeps gamma=0.01 and you expected gamma=0.0001 to win.
Why it fails
The candidates were fitted the same way; only the comparison rule can change the winner.
Do this instead
Check the scoring argument first, then compare the candidates using that same metric.
Common mistake
Writing scoring='accuracy' and expecting it to change the result
What it looks like
Adding scoring='accuracy' to a classifier search and seeing the same winner as before.
Why it fails
Accuracy is already the default for classification, so the explicit argument changes nothing.
Do this instead
Treat the explicit string as documentation, and change the argument only when you actually want a different metric.
Common mistake
Reading a cross-validation score as a score for the whole model across all settings
What it looks like
Reporting the winning CV score as if it were the score of every candidate.
Why it fails
Each candidate has its own fold-averaged score, and only the winner's score is reported as the selected result.
Do this instead
Compare candidates by the same metric, and keep track of each candidate's score when you need to explain a choice.
Common mistake
Assuming AUC is always the better metric
What it looks like
Switching to roc_auc because it sounds more sophisticated, without checking the results.
Why it fails
The source states that the finding is not that AUC is always superior; it changed the winner for one grid and dataset.
Do this instead
Pick the metric that matches the question you need answered, then verify the winner against the test set.

Locating the Cause of a Surprising Winner

MEDIUM

A grid search over one gamma parameter selects a setting you did not expect. Describe the steps you would take, in order, to find out whether the scoring argument is responsible. Explain what you would compare, and why you would compare candidates using the same metric.

Hints

Start with the scoring argument, since it is the step that decides the ranking.
Hold the grid, the training split, and the test split constant while you change only the metric.
Recall that the cross-validation score of the chosen setting should agree with the test set score when the search is behaving normally.

Your attempt (not saved)

Key Takeaways

The scoring argument, passed as a string, sets the metric that cross_val_score and GridSearchCV use to evaluate candidates, and the default for classification is accuracy.
Scoring enters a grid search only at the comparison step, so changing the metric can change which parameter setting is selected while fitting stays the same.
When a selected parameter surprises you, check the scoring argument first, then compare the candidate values using that same metric.
The most important scoring strings include accuracy, roc_auc, average_precision, and the f1 family for classification, and r2, mean_squared_error, and mean_absolute_error for regression.
Error metrics follow a higher-is-better convention in scikit-learn scorers, which is why some appear with a neg_ prefix.

Related concepts

More concepts starting with C