Evaluating Model Performance with Cross-Validation

cross_val_score evaluates a model by splitting data into training and testing sets and returns one score per evaluation run.

Programming

Interactive lab

Try it: K-Fold Cross-Validation and Model Selection

How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.

How it works

Set 10 test points aside; they take no part in any choice.
Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.

Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.

Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.

Educational simulation

Loading the simulation…

One Call That Produces Several Scores

A single accuracy number can be misleading. It depends on which rows happened to land in the testing portion of the data. Cross-validation addresses this by evaluating a model more than once, each time on a different split of the data, so you can see how the model performs across those splits rather than on one arbitrary split.

In scikit-learn, cross_val_score handles this whole process in one call. You do not write a splitting loop yourself. The function splits the data into training and testing sets, fits the model on the training part, scores it on the testing part, and repeats this for each evaluation run. It returns one score per run.

Reading the Evaluation Program

Read the program below once through before looking at any explanation. It loads the iris dataset, creates a LogisticRegression model, and asks scikit-learn to evaluate that model with cross-validation in a single call. Then it prints the result.

python

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

iris = load_iris()
logreg = LogisticRegression()
scores = cross_val_score(logreg, iris.data, iris.target)
print(scores)

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

iris = load_iris()
logreg = LogisticRegression()
scores = cross_val_score(logreg, iris.data, iris.target)
print(scores)

Notes on the code

Line 1 to 3 bring in the dataset loader, the model class, and the evaluation function
Line 5 creates the dataset object that holds the iris data and its labels
Line 6 creates the model object that will be evaluated
Line 7 is the single evaluation call; it returns the value stored in scores
Line 8 prints whatever scores holds

Notice what is absent. There is no loop, no manual split, and no training step written out. Everything between the model and the final printed numbers happens inside cross_val_score. That is why a surprising result is hard to diagnose by staring at the output: the work is hidden inside one line, so the useful question becomes which line produced the surprise.

Following the Values Step by Step

Trace what the program does in order. Each stage depends on the one before it.

1
Line 5 creates the dataset object. At this point iris exists and holds the data and the labels.
2
Line 6 creates the model object. logreg now refers to an unfitted LogisticRegression model.
3
Line 7 makes one call to cross_val_score. The scores name does not exist until this call returns.
4
Line 8 prints scores. The print line can show nothing useful until scores holds a value.

What do you think happens?

Before you look at the printed result, what kind of object is scores, and how many entries does it hold?

Reveal answer

Answer:

A list of three scores, one per evaluation run.

The source's iris example prints a list of three scores rather than a single number. The key point to carry forward is that the result is a collection with one entry per run, not one overall number.

The Three Arguments cross_val_score Receives

The source describes the parameters of cross_val_score as three things, given in this order: the model you want to evaluate, the training data, and the ground-truth labels. In the program above, these map directly onto three names. The model is logreg, the data is iris.data, and the labels are iris.target.

What role does each argument play, and how does each one flow into the scoring call?

Because the order matters, the arguments are not interchangeable. The model comes first, then the data, then the labels. The data and labels are paired: each row of iris.data is matched with the label at the same position in iris.target. Keeping that pairing intact is what lets the function score the model against known answers.

How Each Run Uses a Different Split

On every evaluation run, cross_val_score splits the data into a training set and a testing set. The model learns from the training rows and is scored on the testing rows. The next run uses a different split, so the testing rows change. The source states the core idea directly: the function splits the data into training and testing sets and returns one score per evaluation run.

Which slice of the data is held out for testing on each run, and how does that slice change from one run to the next?

The diagram above is illustrative. It shows the idea of a different held-out slice on each run; the source does not list the exact fold boundaries. What the source does establish is that each run produces one score, so the run-by-run view is what makes the returned list meaningful.

Which rows of the dataset go into the training set and which into the test set for each fold, and do any rows overlap?

This index map is an illustrative split of the 150 iris rows into three blocks of 50. Within any single fold, the training rows and testing rows do not overlap: each row is either held out or used for training in that run. Across folds, the testing rows differ, which is exactly why repeating the evaluation gives a more complete picture than one split.

Verifying the Shape of the Output

The shape of a result is one of the fastest checks you can make. Before trusting a score, ask how many entries the returned object holds. The number of entries should match the number of evaluation runs, and each entry is one score.

python

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

iris = load_iris()
logreg = LogisticRegression()
scores = cross_val_score(logreg, iris.data, iris.target)
print(len(scores))
print(scores)

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

iris = load_iris()
logreg = LogisticRegression()
scores = cross_val_score(logreg, iris.data, iris.target)
print(len(scores))
print(scores)

Notes on the code

len(scores) counts how many entries the returned object holds
Comparing this count with the number of runs is the shape check

Output

The first printed line is the count of entries, which for the iris example described in the source should be 3. The second line is the list of scores itself, one value per run. The exact score values depend on the run and are not reproduced here.

Mistakes That Make Results Look Wrong

Common mistake
Expecting a single number from cross_val_score
What it looks like
Printing scores and expecting one accuracy value, then being confused by a list.
Why it fails
The function returns one score per evaluation run, so the result is a collection.
Do this instead
Read the output as a list. Use the number of entries to confirm how many runs happened.
Common mistake
Passing the arguments in the wrong order
What it looks like
Calling cross_val_score(iris.target, logreg, iris.data) instead of model, data, labels.
Why it fails
The source specifies the order as model, training data, and ground-truth labels, so swapped arguments do not describe the intended evaluation.
Do this instead
Check each argument against its role: the model first, then the data, then the labels.
Common mistake
Blaming the model when an argument is the cause
What it looks like
Assuming LogisticRegression performs badly when the real issue is that the labels were passed in the data position.
Why it fails
Because the whole evaluation happens inside one call, the error is hidden in that line, and the surprise is easy to misattribute.
Do this instead
Trace each named value in the program to find the exact line where expectation and output part ways.

Finding Where an Unexpected Result Begins

When a result surprises you, do not start by changing the model. Walk the program from the top, and at each line ask what value the named variable now holds. The first line whose value differs from what you predicted is where the problem begins. The lines before it were behaving as expected, so the fault lies at or just after that point.

MEDIUM

A colleague's version of the program calls cross_val_score(iris.target, logreg, iris.data) and prints the result. Walk through the lines in order. Which line is the first place the result stops matching what the call was supposed to do, and what is the correct order of the three arguments?

Hints

Compare the argument order in that one call with the model, data, labels order described in the source.
Lines 5 and 6 only create objects; the problem is likely in the evaluation line.

Your attempt (not saved)

What to Carry Forward

1
cross_val_score evaluates a model by splitting data into training and testing sets and returns one score per evaluation run.
2
Its three arguments, in order, are the model, the training data, and the ground-truth labels.
3
The output for the iris example is a list of three scores, so verify the shape by counting entries.
4
When a result looks wrong, trace each named value line by line to find where expectation and output diverge.

Key Takeaways

cross_val_score evaluates a model across several train-and-test splits in a single call, returning one score per evaluation run.
The three arguments are the model, the training data, and the ground-truth labels, and their order matters.
The returned object is a list, and checking its length against the number of runs is a quick shape check.
When a result is unexpected, trace named values line by line to find the exact point where the output departs from the prediction.

Related concepts

Training and Testing Splits in scikit-learn

More concepts starting with E