Class Imbalance in Classification

Most people meet k-fold cross-validation as a recipe. You cut the dataset into k folds, let each fold take a turn as the test set, train on the remaining folds, and average the scores. The recipe says nothing about which samples land in which fold, so it is easy to assume that cutting the data into consecutive pieces is harmless. This assumption breaks down when a dataset is sorted by its labels.

The misconception to watch for is that a standard k-fold split is good enough for evaluating a classifier no matter how the classes are distributed. For a classifier, the class mix inside each test fold matters, and a plain split does not guarantee it. Stratification matters most for imbalanced classification, where a standard split can produce a test fold that says little about the minority class.

A standard k-fold split assigns rows by position. If the data is ordered by class label, the folds can contain only one class. The source uses the iris dataset to show this. Its target labels come in three consecutive blocks: the first third of the samples is class 0, the second third is class 1, and the last third is class 2. Three-fold cross-validation on this ordered data gives each test fold one class only. The first fold's test set holds only class 0, and its training set holds only classes 1 and 2. Every one of the three splits has the same mismatch, so the classes in the training and test sets differ in every split. The source reports that three-fold accuracy on this arrangement would be zero, which is far below what a working classifier achieves on iris. The zero is a symptom of the split, not of the classifier.

The same effect can be seen in a smaller case that is easy to trace by hand. Take twelve samples whose labels are AAAA, then BBBB, then CCCC, and split them into three folds in order. Fold 1 takes positions 0 to 3, fold 2 takes positions 4 to 7, and fold 3 takes positions 8 to 11. Each fold is one contiguous block, so each fold is one class.

Read the diagram by position. The rows are not shuffled before splitting, so the row order that came from the data source decides the fold assignment. Fold 1 receives only class A, fold 2 only class B, and fold 3 only class C. A classifier evaluated on fold 1 is asked to predict a class it never saw in training.

Stratified k-fold cross-validation keeps the proportions between classes the same in each fold as in the whole dataset. The split is still into k folds, and each fold still takes a turn as the test set. What changes is the requirement the folds must satisfy. The folds are built so that each one mirrors the class composition of the full data, rather than whatever happens to sit in a given block of rows.

This formula states the rule. Here n_c is the number of samples of class c in the whole dataset, n is the total number of samples, n_{c,f} is the number of class c samples in fold f, and n_f is the number of samples in fold f. For each class, the share of that class inside a fold should match its share in the full dataset. The approximately-equal sign is deliberate. Real datasets do not always divide evenly, so the goal is that each fold is representative, not that every fold has an identical count.

In the plain split, the 30 samples are taken in order with the class B samples at the end of the data. The first two test folds contain no class B at all, and the third holds all three. In the stratified split, every test fold mirrors the 90 and 10 percent mix. Each fold therefore gives the classifier a chance to be evaluated on the minority class.

Now trace the plain split of the 30-sample dataset from the previous section, iteration by iteration. The divergence happens at the first iteration. Fold 1 is the test block of positions 0 to 9, all class A, so the test fold contains 0 class B samples where the expected count is 1. The mismatch is visible before any training happens, because the composition of the test set is fixed by its position in the data.

The source recommends stratified k-fold cross-validation as the usual choice for evaluating a classifier, because it gives more reliable estimates of generalization performance. Suppose only 10 percent of the samples belong to class B. With standard k-fold, one fold can easily contain only class A. Using that fold as the test set tells you little about overall performance, because the classifier was never asked to handle class B on that split.

Scikit-learn uses stratified k-fold for classification by default. Regression is different. For regression, scikit-learn uses standard k-fold by default. The source notes that one could try to make each fold representative of the different values the regression target takes, but this is not a commonly used strategy and would surprise most users. Stratification is about class proportions, so it applies to classification problems.