Using cross_val_score for model evaluation
A cross-validation splitter passed as the cv parameter gives finer control over how data is split than a plain fold count.
Why a Good Model Can Get a Bad Score
Many learners start cross-validation with a simple belief: a model's score depends only on the model, so as long as the data is split into folds, the evaluation is fair. In practice, the way the data is cut into folds matters just as much as the model. A split that looks neutral can quietly make the evaluation meaningless.
The misconception this article targets is that the default splitting strategy always works. For most use cases, plain k-fold cross-validation for regression and stratified k-fold for classification work well. There are cases where a different strategy is wanted, and one of the most common is reproducing someone else's results with standard, non-stratified k-fold on a classification dataset. Whether that choice is harmful depends on how the samples are ordered, and that is the part learners tend to miss.
What do you think happens?
Suppose the samples are arranged so that all samples of class A come first, then all of class B, then all of class C. If you cut that ordered sequence into three consecutive folds without shuffling or stratifying, what would each fold contain? What would a classifier trained on two of those folds have seen when asked to predict the third?
Reveal answer
Each fold would contain only one class. When one fold is held out for testing, the class it contains is absent from the training data, so the classifier has never seen that class.
The source describes the ordered-label situation exactly this way: each fold corresponds to one class, so when that fold is held out, the held-out class is missing from training.
Passing a Splitter Object as cv
The cv parameter of cross_val_score can change the number of folds when you give it a plain number. scikit-learn goes further: cv can accept a cross-validation splitter, which is an object that controls how the data is split during cross-validation. Instead of passing a number and accepting whatever splitting rule sits behind it, you build the splitter yourself and hand it over. That is what gives you finer control over the splitting step.
The splitter used here is the KFold class, which lives in the model_selection module. KFold splits data into k folds. You import it and instantiate it with the number of folds you want, set through the n_splits argument. The instance is a reusable object, so it can be passed to cross_val_score directly as the cv argument. The call then uses exactly the folds this object produces, rather than building its own.
from sklearn.datasets import load_iris
from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = load_iris(return_X_y=True)
model = DecisionTreeClassifier(random_state=0)
splitter = KFold(n_splits=3)
scores = cross_val_score(model, X, y, cv=splitter)
print(scores)- 1Load the iris features X and labels y.
- 2Create the classifier that will be trained and tested on each fold.
- 3Create the splitter object with three folds and no shuffling.
- 4Run cross-validation: each score comes from training on the other folds and testing on the held-out fold.
- 5Print the list of per-fold scores. If this list looks wrong, the cause is usually which samples landed in which fold.
- The model is any scikit-learn classifier; the splitting rule is the subject of this article.
KFold(n_splits=3)creates a reusable splitter that makes three folds.- Passing the splitter to cv tells
cross_val_scoreto use exactly those folds.
Tracing the Folds in an Unshuffled Split
Consider a small illustrative dataset of twelve samples whose labels are sorted in three blocks of four: rows 0 to 3 are class A, rows 4 to 7 are class B, and rows 8 to 11 are class C. An unshuffled three-fold split cuts this sequence into consecutive blocks of four rows, so each test fold lands on exactly one label block.
Tracing the held-out class in Fold 0
Using the unshuffled three-fold split of the twelve-sample dataset, what does the classifier see when fold 0 is held out for testing?
- 1Identify the test fold
Fold 0 holds rows 0 to 3, all labeled class A.
- 2Identify the training folds
Folds 1 and 2 hold rows 4 to 11, all labeled class B or class C.
- 3Compare the two sides
The training data contains no class A samples at all, while the test fold contains only class A.
- 4Judge the score
A model cannot predict a class it has never been trained on, so a uniformly poor fold score points to the split rather than the model.
Holding out fold 0 means the classifier is tested on a class it never saw during training. The same pattern holds for every fold, which is why the scores can look uniformly wrong.
Key Takeaways
- A cross-validation splitter passed as cv gives finer control over splitting than a plain fold count.
- KFold is configured with
n_splits, and its instance is passed tocross_val_scoreas the cv argument. - On ordered labels, unshuffled folds can each contain only one class, so the held-out class is missing from training and scores can be misleading.
- shuffle=True removes label ordering, and a fixed
random_statemakes the shuffled split reproducible across runs. - When a score looks poor, trace it back to the splitting step: which samples landed in which fold, and whether label ordering was preserved.