Train-Test Splits and Their Limitations

This article starts by running a complete evaluation and then works backward through it. The program below loads the iris dataset, creates a logistic regression model, and asks scikit-learn to evaluate that model with cross-validation. Read the code once, then look at what it prints before reading the explanation that follows.

Before you move on, notice the three objects in the call: the model, the data, and the labels. Those three inputs are all that cross_val_score needs to run the evaluation. The next question is what it hands back.

The general form of the call takes the model to evaluate as its first positional argument, the data (the features) as its second, and the target (the ground-truth labels) as its third. A fourth argument, cv, is optional and controls how many folds are used. When you do not pass cv, the default is three folds.

Cross-validation evaluates a model by splitting the data into folds and then training and testing on different subsets. In K-fold cross-validation, the data is split into k subsets. The model is evaluated once for each fold, so the function produces one score per fold. Each value in the returned array is the result of one held-out fold. The array is therefore a list of per-fold results rather than a single overall score.

Read the diagram as a rule rather than a picture. Each fold is held out once, the model is trained on the remaining folds each time, and the output array holds one accuracy value per held-out fold, in fold order. A single number is not the result of the whole procedure. It is the list of per-fold results.

The cv parameter changes the number of folds. Keep the same model and data, and pass cv=5 instead of relying on the default. The rule from the previous section still applies: the number of folds sets the length of the returned array.

Cross-validation is a repeated train-and-evaluate loop. The same steps run once for every fold, and the loop's output is collected in order. Following one round from start to finish makes the numbers in the returned array easier to interpret.

The loop has no hidden side effects that change the answer. It trains on the training folds, evaluates on the held-out fold, keeps the result, and moves on. When the loop finishes, the collected results are what cross_val_score returns.

When an output does not match your expectation, the useful question is where the divergence began. Work through the chain in order: the inputs you passed, the value of cv, the number of values in the array, and then the values themselves. Checking in this order lets you find the failing step instead of rereading the whole program.

Suppose you have a dataset and you want to build a machine learning model from it. A natural question follows: how will you know whether the model is any good? If you judge the model only on the same data it was built from, you are measuring how well it fits the examples it has already seen. That is not the same as measuring how well it will handle new cases.

A train-test split divides a dataset into a training set and a test set. It is the standard way to keep those two questions apart: how well the model fits known examples, and how well it works on new ones. Keep this question in mind as you read: if the model is supposed to work on new data, which part of the dataset should be used to judge it, and which part should be used to build it?

The two parts have different jobs. The training set is the material the model is built from. The test set is held back and used afterward to check the model's performance on data it was not built from. The test set contains data that is new to the model, which is exactly what makes it useful for judging generalization.

Notice that the test set reaches the evaluation step but does not feed into building the model. That separation is the whole point of the technique. The training set is used to build the model, and the test set is used to evaluate it.