Cross-Validation in scikit-learn

When you build a model, you want to know how well it will perform on data it has never seen. A common first step is to divide your data into a training set, which is used to build the model, and a test set, which is used to measure its accuracy. The trouble is that this measurement depends on which rows happened to land in the test set. A different random split could produce a noticeably different accuracy number. A single split therefore gives you one estimate that may be lucky or unlucky.

Cross-validation addresses this weakness by evaluating the model many times on different splits of the same data. The key idea is that every row in the dataset gets a turn in the test set. No single arrangement of rows decides the final score. Because the evaluation covers the whole dataset across several splits, the resulting picture of performance is more reliable than what any one split can provide. The word generalization points to the goal: you care about how the model will behave on new data, and cross-validation estimates that from the data you already have.

The most commonly used version is k-fold cross-validation, where k is a number you choose, usually 5 or 10. The first step is always the same: the data is partitioned into k parts of approximately equal size. Each part is called a fold. In five-fold cross-validation the data becomes five folds. In the simplest arrangement, the first fifth of the data forms the first fold, the second fifth forms the second fold, and so on.

The word approximately matters. When the number of rows does not divide evenly by k, the folds cannot all be exactly the same size, so they will be as close as possible. The diagram below uses a small ten-row dataset split into five folds of two rows each. The rows are numbered from 0 to 9 for illustration only.

Once the folds exist, a sequence of models is trained, and each model follows the same pattern. One fold is set aside as the test set, and the model is built using the data in all the other folds. The accuracy is then evaluated on the held-out fold and recorded. The next model uses a different fold as its test set, and the process repeats until every fold has been the test set exactly once.

In scikit-learn you do not have to write the splitting loop yourself. The function cross_val_score, which lives in the model_selection module, performs the repeated work and returns one score per split. The code below loads a small dataset and evaluates a LogisticRegression model with five folds. The dataset is an illustrative choice; any feature matrix and label array would work the same way.

The first three inputs map directly onto the three things needed to judge a model. The model is the estimator you want to evaluate. The data is the feature matrix the model learns from. The labels are the correct answers that each prediction is compared against. Because the function receives an unfitted model object rather than a trained one, it can create the fitted copies it needs for each split. The cv argument then decides how many of these rounds run.

The parameter cv sets the number of folds, and the function runs that many rounds and returns that many scores. The idea is the same at any value of k; only the number of rounds changes. A larger k means more folds, so each fold is smaller and more models are trained. A smaller k means fewer, larger folds. The source describes choosing k as a trade-off: a higher number of folds can give a more robust estimate because the score is averaged over more rounds. The source gives no rule for the best value, so treat k as a choice you make and report.

A list of fold scores is hard to act on by itself. A common way to summarize cross-validation accuracy is to take the mean, which gives one number estimating average performance. The mean is the sum of the individual scores divided by how many there are.

The source reads the worked result as meaning the model is expected to be around 96 percent accurate on average. The mean gives a single expectation to report, while the individual scores still hold more detail. The mean, however, hides how much the scores differ from one another. In the source's five-score worked example, the scores range from 100 percent down to 90 percent. The source notes that this relatively high variance between folds could mean the model depends heavily on which folds were used for training. It also notes that the same spread could simply be a consequence of the small size of the dataset. The scores alone cannot tell these two explanations apart, so a wide spread is a prompt for further investigation rather than a conclusion.