Evaluating Classifiers with Cross-Validation

Many people first meet k-fold cross-validation with a reasonable-sounding belief: if you cut a dataset into k pieces, each piece is a fair miniature of the whole. Under that belief, only the number of folds matters, and the order of the rows is a detail you can ignore. This article tests that belief on the iris dataset, which has three classes of flower, and shows where it breaks.

The belief fails most dangerously when the rows are not in random order. Many datasets are stored grouped by label, so all the samples of one class sit together, followed by all the samples of the next class. When you split such a dataset into consecutive chunks, each chunk may contain only a slice of the classes rather than a sample of all of them.

The simple strategy of taking the first third, the second third, and the last third treats row position as if it carried no information. But in a label-ordered dataset, position is the label. Splitting by position turns the label ordering into a fold structure: each fold inherits whatever classes happen to occupy its position. The folds end up holding very different mixtures of classes, and in the iris case each fold is a single class, so no fold can represent the whole dataset.

Trace what this means for evaluation. When fold 1 is the test set, the training set contains only classes 1 and 2, so the classifier has never seen class 0. Every test sample belongs to a class absent from training, so every prediction is wrong. The same thing happens in the other rounds. The accuracy collapses not because the classifier is weak but because the test data and the training data share no classes.

Stratified k-fold cross-validation fixes this by building each fold so that the proportions between classes match the proportions in the full dataset. Under standard splitting on ordered labels, each fold is one class. Under stratified splitting, each fold is built so that its class proportions match those of the full dataset. The difference is not in how many folds exist or how many samples each fold holds. It is in what each fold is allowed to contain.

The key word is proportions. Stratification does not promise that every fold has the same number of samples, and it does not change the rule that each fold takes a turn as the test set. It promises that whatever mix of classes exists in the full dataset is reproduced, as closely as the counts allow, inside every fold. The source describes the same idea with a 90 and 10 percent split: in each fold, 90 percent of the samples belong to class A and 10 percent belong to class B.

The next example uses a dataset we construct for illustration. It has 100 samples, 90 of class A and 10 of class B, stored with all class A rows first. Five folds means 20 samples per fold. The diagram below compares the class counts in three of the five folds under plain consecutive splitting and under stratified splitting.

Under consecutive splitting, folds 1 through 4 hold only class A, so a test fold among them contains no class B examples at all. Fold 5 holds the class B samples and no class B samples appear elsewhere in its position. Under stratified splitting, every fold holds 18 class A and 2 class B samples, which is the same 90 to 10 ratio as the whole dataset. Now every test fold gives some information about how the classifier handles class B.

You can predict the class counts in each fold before running any code. If a class has $n_c$ samples in the full dataset and you use $k$ folds, stratification aims for each fold to hold about $\frac{n_c}{k}$ samples of that class. The counts only match exactly when $n_c$ divides evenly by $k$; otherwise the fold sizes differ by at most one sample, as closely as the counts allow.

Stratified and standard k-fold share the same rotation. In each of k rounds, one fold serves as the test set and the remaining folds form the training set. Each fold, used in turn as the test set, produces one estimate, and the estimates are combined. Because every fold takes exactly one turn as the test set, every sample is tested exactly once.

Stratification changes what is inside each fold, not this rotation. If the test fold in round 2 is stratified, it carries its share of every class, so the estimate from that round reflects all classes. This is why the combined result is more trustworthy: each round's estimate comes from a test set that represents the dataset.

The following mistakes come up often when people first apply cross-validation to classifiers.

For evaluating a classifier, stratified k-fold usually gives more reliable estimates of generalization performance, especially when some classes are rare. Standard k-fold has a different default role in scikit-learn. The comparison below summarizes the two strategies.

Scikit-learn uses standard k-fold cross-validation by default for regression. The source notes that one could try to make each fold representative of the different values a regression target takes, but this is not a commonly used strategy and would surprise most users. Stratification is defined in terms of class labels, so it applies to classification, where it is the usual choice for evaluating a classifier.