Stratified Cross-Validation for Classification

Cross-validation is usually treated as a safe way to evaluate a model. You split the data into folds, train on some folds, test on the remaining one, and repeat until every fold has been the test set once. A common assumption is that this procedure is robust to how the rows happen to be arranged. If there are enough folds, the thinking goes, the evaluation should be fair no matter what order the samples came in.

The source shows that this assumption is false. When the samples are ordered by label, a plain split can hand each fold a single class. The model is then evaluated on data it has never learned to recognize, and the resulting scores tell you very little about how the model would behave on new data. This article follows one such case step by step, then introduces the tools scikit-learn provides to take back control of how the data is divided.

The source is also direct about the usual situation. For most problems the defaults work well: regression uses standard k-fold, and classification uses stratified k-fold. The problem described here is not caused by those defaults. It appears when you have a concrete reason to choose a different strategy, such as reproducing someone else's published experiment that used standard k-fold on a classification dataset.

Before reading the trace, commit to a prediction. The iris dataset, as used in the source, is ordered so that each class occupies a contiguous block of rows. Suppose you split it into three folds with KFold, leave shuffling off, and use logistic regression as the model. What do you expect the three fold scores to look like?

The mechanism that gives you control is the cross-validation splitter. A splitter is an object that controls how the data is split during cross-validation. You pass it directly as the cv parameter to cross_val_score. Instead of asking for a number of folds, you hand over a configured splitter, and that splitter decides the folds itself. The number of folds can still be set this way, through the n_splits argument of KFold.

The source uses a deliberately bad configuration to check whether the splitter behaves as expected: three folds, no stratification, no shuffling, on a classification dataset ordered by label. The goal is to verify the claim that three-fold non-stratified cross-validation on iris is a very poor choice. Follow the rows in order.

Plain KFold splits by position. It takes the first block of consecutive rows as fold one, the next block as fold two, and the last block as fold three. Iris has 150 rows, with the classes stored as three contiguous blocks. Cutting those 150 rows into three consecutive pieces lines up each piece with one class, which is the exact outcome the source describes.

Now look at what each round trains on. When fold one is the test set, the model is trained only on the versicolor and virginica rows. When fold two is held out, it is trained on setosa and virginica. Each round, the held-out fold is a class the model has never seen during training. That is the exact point where the result departs from a reasonable expectation: a fair evaluation needs each class to be represented in the training data, and here no fold provides that.

Plain KFold does not look at labels at all. It only counts positions. That is why the fold boundaries say nothing about the classes they contain. When the rows are already sorted by label, the boundaries and the classes line up, and the scores stop measuring how well the model generalizes.

The source offers a remedy that does not require stratification: shuffle the data before splitting it. Shuffling removes the ordering of the samples by label. Shuffling is switched on with the shuffle parameter of KFold. Passing shuffle=True randomly reorders the samples before they are cut into folds, so each fold is likely to contain a mix of classes. The source reports that shuffling before splitting yields a much better result than the unshuffled run.

Shuffled and unshuffled runs differ in exactly one respect: whether the samples are reordered before they are split. Everything else in the setup can stay the same. That single change is what moves the class mix of each fold away from one class per fold.

Shuffling introduces randomness, and randomness introduces variation between runs. Without a fixed seed, each run draws a different random ordering, which means each run creates different folds and therefore different scores. Fixing random_state makes the shuffle repeatable: the same seed produces the same fold assignment every time the code runs.

Shuffling is one remedy. Stratified splitting is another. The source's default for classification is stratified k-fold, and it differs from plain KFold because it looks at the labels. Plain KFold only cares about position. A stratified splitter arranges the folds so that each fold keeps roughly the same class proportions as the full dataset, so no fold ends up with one class in place of the others.

The reason this matters for the score is that a test fold containing a class the training folds never saw cannot be classified correctly by a model that never learned that class. Stratification and shuffling both address this by making each training set resemble the full dataset, but they reach that goal differently. Stratification uses the labels directly, while shuffling relies on the random reordering to break up the class blocks.