Measuring Model Accuracy

When you build a model, the question that matters most is how well it will perform on data it has never seen. A model that scores well on the data it was built from may still fail on new cases, so the training data alone cannot answer the question. You need a way to measure performance on held-out data, meaning rows the model did not learn from.

A common first step is to divide the data into two parts. One part is the training set, used to build the model. The other is the test set, used to measure its accuracy. This approach is sound in principle, but it hides a weakness: the measured accuracy depends on which rows happened to land in the test set. Keep that weakness in mind as you work through this article, because it is the problem cross-validation was designed to solve.

The separation is what makes the accuracy number meaningful. If the test rows were also used for building the model, the score would describe how well the model memorized them rather than how it would behave on new data. Whichever method you use, the rule stays the same: the rows that measure accuracy must stay out of the building step for that model.

A different random split could produce a noticeably different accuracy number. Suppose two people split the same dataset at random. Each person's test set contains different rows, so each person's model is measured on a different sample of the data. One split might happen to include mostly easy cases and report a high accuracy. Another might include mostly difficult cases and report a low one. A single split gives you one estimate, and that estimate may be lucky or unlucky.

The diagram shows the contrast. A single split leaves you with one number and no way to see how much it would move under a different arrangement of rows. Repeating the evaluation over several splits produces several numbers, so the spread of results becomes visible. The key idea of cross-validation is that every row gets a turn in the test set, so no single arrangement of rows decides the final score.

In the simplest arrangement, the first fifth of the data forms the first fold, the second fifth forms the second fold, and so on. The diagram below uses a generated example of ten rows divided into five folds of two rows each. Real datasets rarely divide so neatly.

The word approximately matters. When the number of rows does not divide evenly by k, the folds cannot all be exactly the same size, so they are made as close as possible. For example, if eleven rows were split into five folds, four folds would hold two rows and one fold would hold three. The method does not change because of this uneven size. Each fold still serves as the test set exactly once.

The letter k controls the trade-off. A larger k means more folds, so each fold is smaller and more models are trained. A smaller k means fewer, larger folds. The procedure itself, which is partition, rotate the test fold, train, score, and collect the results, stays the same for every value of k.

Once the folds exist, a sequence of models is trained, and each one follows the same pattern. One fold is set aside as the test set. The model is built using the data in all the other folds. Its accuracy is then evaluated on the held-out fold, and that accuracy is recorded. The next model uses a different fold as its test set, and the process repeats until every fold has been the test set exactly once.

Read the diagram row by row. In each round, exactly one fold is the test set and the other four folds together form the training set. The test fold changes from round to round, so each row is held out exactly once. The trained model in each round is evaluated only on its own held-out fold, which means no round scores a model on rows that model was built from.