Cross-Validation

Many learners first meet cross-validation as a more careful version of splitting data once. It runs the model several times, and it is easy to conclude that the end result is a stronger model, ready to be used on new data. This article looks at what cross-validation actually gives you, why it is valuable, and where its limits lie. The first step is to name the belief that causes the most trouble.

When you build a model, you want to know how well it will perform on data it has never seen. A common first step is to divide your data into a training set, used to build the model, and a test set, used to measure its accuracy. That measurement depends on which rows happened to land in the test set. A different random split could produce a noticeably different accuracy number, so a single split gives you one estimate that may be lucky or unlucky.

The function train_test_split shuffles the data and divides it, so the outcome depends on which examples landed on each side. Picture two draws. In one, all the difficult-to-classify examples go into the training set. The test set then contains only easy cases, and the accuracy looks impressively high. In the other, the difficult examples all land in the test set, and the accuracy looks disappointingly low. In both cases the score tells you more about the draw than about the algorithm.

The most commonly used version is k-fold cross-validation, where k is a number chosen by the user, usually 5 or 10. The first step is always the same: the data is partitioned into k parts of approximately equal size. Each part is called a fold. In five-fold cross-validation, the data becomes five folds. In the simplest arrangement, the first fifth of the data forms the first fold, the second fifth forms the second fold, and so on.

The word approximately matters. When the number of rows does not divide evenly by k, the folds cannot all be exactly the same size, so they are made as close as possible. Once the folds exist, a sequence of models is trained. Each model follows the same pattern: one fold is set aside as the test set, the model is built from the data in all the other folds, the accuracy is measured on the held-out fold, and that accuracy is recorded.

The key idea is that every row gets a turn in the test set. No single arrangement of rows decides the final score. The diagram below shows how five-fold cross-validation moves through the data. Each round holds out one fold for testing and trains on the remaining four.

Notice what the rotation guarantees. Each fold appears in the test role exactly once, so every example is tested exactly once. Across the five rounds, every example also appears in the training role four times, because it is used for fitting in every round except the one where it is held out.

Five-fold cross-validation trains five models and collects five accuracy values. The rounds run in sequence, and each round produces exactly one accuracy value. The diagram below traces that progression from the first round to the final result.

The choice of k is set by the user. A larger k means more folds, so each fold is smaller and more models are trained. A smaller k means fewer, larger folds. The method itself stays the same in both cases: partition, rotate the test fold, train, score, and collect the results.

A typical single split reserves about 25 percent of the data for evaluation and trains on the remaining 75 percent. With cross-validation, each iteration trains on k minus 1 of the k folds, so the training share per model is $\frac{k-1}{k}$ of the dataset. The test share for each round is $\frac{1}{k}$.

Ten-fold evaluation therefore lets each model fit on more of the data than five-fold evaluation does, at the price of more training runs. The gain is in how efficiently the data is used, not in any change to what the final score means.

Cross-validation evaluates generalization performance by splitting data repeatedly and training multiple models. That is more stable than a single train-test split, because the final picture is built from several different arrangements of the same rows rather than one arrangement. The diagram below compares the two approaches side by side.

The spread of scores across folds is itself informative. The source reports that five-fold cross-validation on the iris dataset produced accuracies anywhere between 90 and 100 percent. That spread shows how sensitive the model is to which examples it trains on, and it gives a sense of the worst and best cases you might see when the model meets new data.

Cross-validation answers a narrower question than the one learners often have in mind. It asks how well a particular algorithm, trained on a particular dataset, is likely to generalize. The answer comes from repeatedly training on part of the data and scoring on the part that was held back. The models built during the process are discarded once their scores are recorded.

Every benefit above has a price. The main disadvantage of cross-validation is increased computational cost. Because it trains k models instead of one, cross-validation is roughly k times slower than a single split of the data. A five-fold evaluation therefore takes about five times the training work, and a ten-fold evaluation about ten times.

The extra work comes entirely from repetition. Each round is a full training run on a different set of folds, so the cost grows with the number of rounds you choose to run.