k-Fold Cross-Validation

Most people first meet k-fold cross-validation as a recipe. You cut the dataset into k folds, let each fold take a turn as the test set, train on the remaining folds, and average the scores. The recipe itself says nothing about which samples land in which fold, so it is easy to assume that cutting the data into consecutive pieces is harmless. This assumption breaks down when the dataset is sorted by its labels.

The misconception to watch for is this: a standard k-fold split is good enough for evaluating a classifier no matter how the classes are distributed. For a classifier, the class mix inside each test fold matters, and a plain split does not guarantee it. In this article you will trace exactly where that guarantee fails, and then see how stratified k-fold cross-validation changes what the split is required to preserve.

Consider a small dataset whose labels come in three consecutive blocks, similar to the iris dataset in the source. The first third of the samples is class 0, the second third is class 1, and the last third is class 2. Here is a version of that label arrangement with 18 samples, six per class, split into three folds with no shuffling. Read the code first, then work out the answer before you check the output.

The index map shows the root of the problem. Fold 1 contains only class 0, fold 2 contains only class 1, and fold 3 contains only class 2. The output confirms the same thing from the other direction: whenever a fold is the test set, the training set contains the other two classes and never sees the class being tested.

Stratified k-fold cross-validation fixes the problem by changing what the split is required to preserve. The split is still into k folds, and each fold still takes a turn as the test set. The difference is the rule that governs how samples are assigned. Each fold must mirror the class composition of the full dataset, rather than whatever happens to sit in a given block of rows.

The source gives a simple ratio to make the rule concrete. If 90% of the samples belong to class A and 10% belong to class B, stratified cross-validation ensures that in each fold 90% of the samples are class A and 10% are class B. The composition of every fold is anchored to the composition of the whole dataset, which is the property a fair test set needs.

In this statement, $n_{c}$ is the number of samples of class c in the full dataset, $N$ is the total number of samples, $n_{c,\text{fold}}$ is the number of class c samples in one fold, and $n_{\text{fold}}$ is the number of samples in that fold. The approximately-equal sign is deliberate. With real datasets the counts cannot always divide evenly, so the goal is that each fold is representative, not that every fold has an identical count.

Why does this matter for evaluation? Suppose only 10% of the samples belong to class B. With a standard k-fold split, one fold can easily contain only samples of class A. Using that fold as the test set tells you little about overall performance, because the classifier was never asked to handle class B on that split. A stratified split guarantees that every test fold contains a fair share of class B, so each round asks the classifier the same kind of question.

Now run the same 18-sample, ordered label arrangement through a stratified splitter. The only change in the code is the class name. The labels are still sorted by class, but the stratified splitter assigns samples so that each class is shared across the folds.

Compare the two outputs. The plain split produced test folds that each contained one class and none of the others. The stratified split produced test folds whose class mix matches the full dataset, which here is an equal three-way mix. The divergence you traced earlier, where the plain split drifted away from the expected mix in its very first round, is gone. The row order of the data has not changed; what changed is the rule that assigns samples to folds.

The source recommends stratified k-fold cross-validation as the usual choice for evaluating a classifier, because it results in more reliable estimates of generalization performance. Stratification is about class proportions, so it applies to classification. Scikit-learn uses stratified k-fold for classification by default and standard k-fold for regression by default.

For regression, the target is a continuous value rather than a set of classes, so there are no class proportions to preserve in the same way. The source notes that it would be possible to try to make each fold representative of the different values the regression target takes, but that this is not a commonly used strategy and would surprise most users. The practical lesson is to match the splitting strategy to the kind of prediction being evaluated.