k-Fold Cross-Validation Procedure

Suppose you have a labeled dataset and want to know how well a model will handle new data. The most common first step is to divide the dataset once, fit the model on one portion, and score it on the portion that was held back. That single score feels like a clean answer. This article asks a sharper question: how much should you trust that one number, and what does a different evaluation strategy buy you? Cross-validation is the answer most practitioners reach for, and its advantages over a single split fall into three groups that we will take in turn.

We will follow the procedure step by step, watching how the held-out portion of the data changes from round to round and how the collection of scores changes as a result. Keep one question in mind throughout: what is the score actually measuring?

A single train-test split is produced by random selection. That randomness is harmless on average, but any one particular draw can be favorable or unfavorable. Think about what happens when the examples that are hardest to classify all land in the training portion. The held-out portion then contains only the comparatively easy cases, and the score you measure looks better than the model deserves. Now reverse the situation: if all the hard cases land in the held-out portion, the score looks worse than the model deserves. In both cases the number is a product of the luck of the draw as much as the quality of the model.

Cross-validation attacks the problem of luck by changing the structure of the evaluation rather than hoping one split is representative. The dataset is partitioned into a number of folds of roughly equal size. In each iteration, one fold is set aside as the test set and the model is fitted on the remaining folds. The procedure repeats until every fold has served as the test set exactly once. Because each example belongs to exactly one fold, each example is evaluated exactly once. No record can escape evaluation through an unlucky or convenient draw.

Notice what the diagram makes visible. The training portion changes in every round because a different fold is removed from it, while the test portion always rotates to a fold that has not yet been held out. After five rounds, the held-out folds cover the whole dataset with no overlap.

Notice that the held-out examples are never shared between rounds. That is what makes the evaluation complete: no single lucky draw can decide the outcome, because every example contributes its turn in the test set.

This changes what a good score means. Each cross-validation score comes from a different held-out fold, so the model has to perform well on every portion of the data, not just on one favored portion. When the individual scores are consistently high, and their mean is high, you have evidence that the model generalizes across the dataset rather than merely fitting one particular split.

The left column shows why a single split is fragile: whichever draw you happen to get determines the answer. The right column shows the alternative. Each fold contributes one score, and the mean of those scores is far less dependent on any one draw, because an extreme draw in one round is balanced by ordinary draws in the others.

Having several evaluations does more than produce a steadier average. The collection of scores also reveals how sensitive the model is to which examples it is trained on. If every fold gives nearly the same score, the model's performance appears stable across these training choices. If the scores vary widely, the model's performance depends noticeably on the particular training data it receives.

A concrete case from the source makes this clear. For the iris dataset, the cross-validation accuracies ranged between 90% and 100%. That is a wide band. Read as a single number, the performance could look excellent or merely acceptable depending on which fold you looked at. Read as a range, the band tells you something useful: it gives an idea of how the model might perform in a worst-case and a best-case scenario when it meets new data. A single split would have hidden this entire picture behind one value.

The third advantage concerns how much of the dataset actually shapes the model. A common single split reserves about a quarter of the data for evaluation, so roughly 75% of the examples are used to fit the model. Cross-validation changes that proportion. In k-fold cross-validation, each iteration fits the model on all folds except one, so the training share is the remaining $k-1$ folds out of $k$ total folds.

The source also notes the practical reason this matters: more data will usually result in more accurate models, so a larger training share is often worth having. Larger k therefore buys more training data per round, but it also means more rounds to run, so the choice of k is a trade-off rather than a rule that bigger is always better.