Cross-Validation with cross_val_score

cross_val_score evaluates a model with cross-validation and takes the model, the data, and the target as its main parameters.

Programming

Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

Set 10 test points aside; they take no part in any choice.
Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

Running a Complete Evaluation First

This article starts with code you can trace. The program below loads the iris dataset, creates a logistic regression model, and asks scikit-learn to evaluate that model with cross-validation using leave-one-out splitting. Read the code once. The rest of the article explains how the numbers it prints are produced and what each one means.

python

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, LeaveOneOut

X, y = load_iris(return_X_y=True)
model = LogisticRegression()
loo = LeaveOneOut()
scores = cross_val_score(model, X, y, cv=loo)

print(len(scores))
print(scores.mean())

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, LeaveOneOut

X, y = load_iris(return_X_y=True)
model = LogisticRegression()
loo = LeaveOneOut()
scores = cross_val_score(model, X, y, cv=loo)

print(len(scores))
print(scores.mean())

Walk through it

1
Line 1 to 3 import the dataset loader, the model, and the cross-validation tools.
2
Line 5 loads the features and labels as two separate objects.
3
Line 6 creates the logistic regression model that will be evaluated.
4
Line 7 creates the leave-one-out splitter and stores it in loo.
5
Line 8 runs the evaluation and stores the returned array in scores.
6
Line 10 and 11 print the number of results and their mean.

Notes on the code

X holds the features and y holds the labels for the iris dataset
loo is the splitter object, created once and stored in a variable
cross_val_score returns one result per split in the array scores
len(scores) counts those results; scores.mean() averages them

The source for this run reports two facts. The number of iterations matched the 150 samples of the iris dataset, and the mean accuracy was 0.95. Before reading further, try to predict the first printed number yourself. Then check your guess against the explanation in the next sections.

What do you think happens?

When this program runs on the iris dataset, what number does print(len(scores)) show?

Reveal answer

Answer:

150

Leave-one-out creates one split for each sample, and the iris dataset has 150 samples. cross_val_score returns one accuracy per split, so the array holds 150 values.

The Three Inputs and the Optional cv Argument

cross_val_score evaluates a model with cross-validation. Its three main positional arguments are the model to evaluate, the data (the features), and the target (the ground-truth labels). Those three inputs are all the function needs to run an evaluation. The cv argument is optional and controls how many folds are used.

python

cross_val_score(model, X, y)
cross_val_score(model, X, y, cv=5)
cross_val_score(model, X, y, cv=LeaveOneOut())

cross_val_score(model, X, y)
cross_val_score(model, X, y, cv=5)
cross_val_score(model, X, y, cv=LeaveOneOut())

Notes on the code

The first call uses the default cv, which is three-fold cross-validation
The second call sets the number of folds explicitly to five
The third call passes a splitter object instead of a number

Folds and the Returned Array

Cross-validation evaluates a model by splitting the data into folds, then training and testing on different subsets. In K-fold cross-validation, the data is split into k subsets. The model is evaluated once for each fold, so the function produces one score per fold. A single number is therefore not the whole result. The result is a list of per-fold results.

When you do not set cv, the function performs three-fold cross-validation by default and returns three accuracy values, one for each fold. The cv parameter sets the number of folds, and the length of the returned array matches that number. Each value comes from one held-out fold, and the values appear in fold order.

Which sample indices land in which fold. This illustrative split uses six samples and three folds, so each fold holds two samples. The size of each fold follows from the number of samples divided by the number of folds.

The diagram above shows the three-fold case. Each fold is held out once, and the model is trained on the remaining folds each time. The array printed by the program therefore holds one accuracy value for each held-out fold, in fold order. Keep this picture in mind: the array is a list of per-fold results, not one overall score.

Formula, step by step

1
$s_i$ is the accuracy from held-out fold $i$, one value per fold in the returned array
2
$k$ is the number of folds, which equals the length of the array
3
The sum adds every per-fold accuracy together
4
Dividing by $k$ gives the mean, which is the single number scores.mean() reports

Tracing Leave-One-Out Through the Splits

Leave-one-out cross-validation is k-fold cross-validation where each fold is a single sample, and each split uses one point as the test set. The splitter LeaveOneOut() is created once and stored in loo. cross_val_score receives loo through its cv argument and performs one fit and one evaluation per split. Because each test set contains exactly one sample, every split holds out one point and trains on the rest.

In each iteration one sample is held out for testing and the remaining samples are used for training. This illustrative run uses four samples. The held-out sample changes on every iteration until each sample has been the test point once.

Worked example

Counting the iterations for the iris dataset

The iris dataset has 150 samples. How many splits does leave-one-out create, and how many training samples does each split use?

1
Count the samples
The iris dataset contains 150 samples, as the source run confirms.
2
Count the splits
Leave-one-out makes each sample the single test point once, so there is one split per sample: 150 splits.
3
Count the training samples
Each split holds out one point, so each training set has 150 minus 1 samples, which is 149.
4
Check against the output
The printed length should be 150. If it is not, the splitting stage is the first place to look.

Result

Leave-one-out creates 150 splits, and each split trains on 149 samples and tests on 1.

Follow the path a single run takes. The scores variable holds one result per split. Printing len(scores) counts those results, which is the iteration count. scores.mean() averages them into the single number that is usually reported. In the source run, the iteration count matched the 150 samples and the mean accuracy was 0.95.

cross_val_score takes the model, the features, and the labels, produces one score per split, and the mean of those scores is the reported number.

Changing the cv Parameter

The cv parameter changes the number of folds, while the model and data stay the same. Suppose the call keeps the same model and the iris data but passes cv=5. The length of the returned array follows the rule from earlier: it matches the number of folds. The size of each test fold also follows from the number of samples divided by the number of folds. The table compares the three settings for 150 samples. The per-fold sizes assume the samples divide evenly across folds, which they do for 150 samples with these settings.

Setting	Values returned	Test samples per split	Training samples per split
Default, three-fold (cv=3)	3	50	100
Five-fold (cv=5)	5	30	120
Leave-one-out (cv=LeaveOneOut())	150	1	149

Effect of the cv setting on a 150-sample dataset

What do you think happens?

If you change the call to cv=10 on the 150-sample iris dataset, how many values does cross_val_score return, and how many samples are in each test fold?

Reveal answer

Answer:

10 values, with 15 test samples in each fold

The number of returned values matches the number of folds, which is 10. Each fold holds 150 divided by 10, which is 15 samples, and each training set holds the other 135.

Advantage and Cost of Leave-One-Out

The main advantage of leave-one-out is that it uses every sample as a test point exactly once. The source describes this as sometimes giving better estimates on small datasets. Its main disadvantage is time. Because the model is trained and tested once per data point, a large dataset means a very large number of fits. Weigh these two points against your dataset size before choosing this method.

Finding Where a Result Goes Wrong

When a leave-one-out result looks wrong, the most useful question is which stage of the process produced the unexpected value. Work through the chain in order: the inputs you passed, the value of cv, the number of values in the array, and then the values themselves. Checking in this order lets you find the failing step instead of rereading the whole program.

The printed iteration count checks the splitting stage. If it does not match the number of samples you expected, the split structure is different from what you assumed. If the iteration count is right but the mean accuracy is unexpected, the problem lies in the model fit or the evaluation for individual splits, not in how the data was divided. A length mismatch usually points to the cv argument. A value mismatch with the correct length points to the data or the model.

Common mistake
Assuming cross_val_score returns one overall score
What it looks like
Printing scores and expecting a single accuracy value.
Why it fails
The function produces one score per fold, so the result is a list of per-fold results.
Do this instead
Read the array, then use scores.mean() when you want the single summary number.
Common mistake
Expecting three values after changing the number of folds
What it looks like
Passing cv=10 and still expecting three values back.
Why it fails
Three is only the default. The cv parameter sets the number of folds, and the array length matches it.
Do this instead
Predict the length from the cv value before running the code, then compare it with len(scores).
Common mistake
Forgetting that leave-one-out creates one split per sample
What it looks like
Expecting a 150-sample dataset to give only a handful of splits with LeaveOneOut().
Why it fails
Each sample becomes its own test set, so the split count equals the number of samples.
Do this instead
Count the samples first. For iris, the expected iteration count is 150.
Common mistake
Rereading the whole program when the output is wrong
What it looks like
Checking the model code line by line when the printed length is already off.
Why it fails
A length mismatch is usually found at the splitting stage, so the rest of the program is not the first place to look.
Do this instead
Check the iteration count first, then the scores and their mean.

Practice Predicting and Diagnosing

EASY

Without running any code, predict the printed value of len(scores) for each call on the 150-sample iris dataset: (a) cross_val_score(model, X, y), (b) cross_val_score(model, X, y, cv=5), (c) cross_val_score(model, X, y, cv=LeaveOneOut()). Then explain, in one sentence each, which part of the call determines the answer.

Hints

Start with the default: when cv is not set, the number of folds is three.
The length of the returned array equals the number of folds.
For leave-one-out, count how many samples become test points.

Your attempt (not saved)

MEDIUM

A colleague runs leave-one-out on the 150-sample iris dataset and reports 150 values in the array, but a mean accuracy that seems too low for the model. Using the diagnostic order from this article, which stage would you check first, which stage would you check second, and why does the correct iteration count narrow down the cause?