Introduction to Model Evaluation with scikit-learn

This article starts by running a complete evaluation before explaining any of its parts. The program loads the iris dataset, creates a logistic regression model, and asks scikit-learn to evaluate that model with cross-validation. Read the program once, then look at the line it prints. The rest of the article explains how that line is produced and what each number in it means.

Before moving on, notice the three objects in the call: the model, the data, and the labels. The general form of the call takes these as its first three positional arguments. The first is the model to evaluate, the second is the data (the features), and the third is the target (the ground-truth labels). Those three inputs are all that cross_val_score needs to run the evaluation. The next question is what it hands back.

Cross-validation evaluates a model by splitting the data into folds and then training and testing on different subsets. In K-fold cross-validation, the data is split into k subsets. The model is evaluated once for each fold, so the function produces one score per fold. The default for cross_val_score is three-fold cross-validation, which means the data is divided into three subsets.

Each fold holds out one subset as test data and trains on the remaining subsets. The held-out subset changes from fold to fold, so every subset is held out exactly once. The diagram below shows the three-fold case. Read it as a rotation: the same three blocks appear in every fold, but a different block is the test block each time.

In the fold where a block is held out, the model is scored on that block. In every other fold, the same block is part of the training data. Because each block is the test block exactly once, the three scores together cover the whole dataset rather than resting on one arbitrary split.

Within a single fold, the work follows a fixed sequence: the model is fit on the training rows, it predicts on the held-out rows, and the predictions are scored as accuracy. Then the process moves to the next fold and repeats. The diagram traces one pass through this cycle.

The loop runs once per fold. With three folds, the sequence runs three times, and each run produces one accuracy value. The model is retrained from the training rows in each fold, so the three results are independent evaluations of the same model on different held-out blocks.

The value cross_val_score returns is an array, and each element is the result of one held-out fold. A single number is not the result of the whole procedure. The array is a list of per-fold results, and its positions follow fold order: the first element comes from the first fold, the second from the second fold, and so on.

The cv parameter sets the number of folds, and the length of the returned array matches that number. The next example keeps the same model and data and passes cv=5. Predict the length of the array from the rule above before you run it: the array should contain five values, one for each of the five folds.

When an output does not match your expectation, the useful question is where the divergence began. Work through the chain in order: the inputs you passed, the value of cv, the number of values in the array, and then the values themselves. Checking in this order lets you find the failing step instead of rereading the whole program.

A mismatch in length usually points to the cv argument. A mismatch in the values, with the correct length, points to the data or the model, because the same fold structure was used. This split tells you which part of the call to inspect first.