Stratified k-Fold Cross-Validation and Other Strategies

Cross-validation is one of the most common ways to check how well a learning algorithm is likely to perform on data it has never seen. It splits the available data into several parts, trains and tests repeatedly, and averages the results. Because it is so central to model evaluation, it is easy to slide into a belief that sounds reasonable but is wrong: that running cross-validation is a way of building the model you will eventually deploy. This article examines that belief, explains what cross-validation actually gives you, and then looks at its main practical drawback, which is cost.

The procedure begins by dividing the available data into k parts, called folds. In each round, one fold is held out as the test set and the remaining k minus 1 folds form the training set. A model is trained on the training folds and scored on the held-out fold. Then the next round holds out a different fold, and this continues until every fold has served as the test set exactly once.

Read the diagram as a rotation. The held-out slice moves one position each round, while the rest of the data serves as training material. Because every fold gets its turn as the test set, every observation contributes to evaluation exactly once, and no single lucky or unlucky split decides the outcome.

Each round of cross-validation does build a model internally, since each fold needs a model trained on its training portion and scored on its held-out portion. The loop below shows that the same train-and-evaluate step runs once per fold, and each run produces one score.

The purpose of cross-validation is to answer a question about an algorithm, not about a particular fitted model. The question is how well a given algorithm will generalize when it is trained on a specific dataset. The internal models built along the way are used only to produce scores. The procedure hands back the evaluation, and those models are not the output.

Suppose the scores look good. The natural next step is to decide whether to use the algorithm, and if so, to train a final model on the full dataset for deployment. Cross-validation supports that decision, but the deployable model still has to be produced by a separate training step. Treating the cross-validation output as if it were the finished product leads directly to the misconception.

The main disadvantage of cross-validation is increased computational cost. A single split of the data trains one model. Cross-validation trains k models instead, one for each of the k rounds, so the total work grows with k. The practical consequence is that cross-validation will be roughly k times slower than a single split of the data.

Here $T_{split}$ is the time to train and score one model on a single split, and $T_{CV}$ is the time for the full cross-validation run. The relation is a rule of thumb that follows directly from the fact that k models are trained where one model would otherwise be trained. The word roughly matters: real running times depend on the algorithm, the size of the data, and the hardware, so the factor of k is a planning estimate rather than an exact measurement.

The title of this article names stratified k-fold cross-validation. It follows the same procedure described above: data is divided into k folds, each fold takes a turn as the held-out test set, and the scores are collected for evaluation. The stratified version differs in how the folds are formed. It aims to keep the mix of classes in each fold close to the mix in the full dataset, which a plain random split does not guarantee. The rotation, the repeated training, and the fact that the output is an evaluation rather than a deployable model all stay the same. The source material for this article does not detail other named variants, so they are not described here.