Summarizing Cross-Validation Scores

Cross-validation gives you a way to judge how well a model is likely to perform by testing it on several different slices of your data. In scikit-learn, the function cross_val_score runs that whole process for you. You hand it a model, the data, and the labels, and it returns the evaluation results. This article starts with one complete run, then explains how the printed output is produced, what each number means, and how the same ideas extend to tuning a model with a grid search.

Read the program below once before running it. It loads the iris dataset, creates a logistic regression model, and asks scikit-learn to evaluate that model with cross-validation. Then look at what it prints. The rest of the article explains how that printed line is produced.

Before you run it, notice the three objects in the call: the model, the data, and the labels. Those three inputs are all cross_val_score needs to carry out the evaluation. The question to ask next is what the function hands back.

The first three positional arguments of cross_val_score are the model to evaluate, the data (the features), and the target (the ground-truth labels). The model is whatever estimator you want to test. The data is the feature table, and the target holds the correct answer for each row. The function uses these three pieces to repeatedly train the model on some portion of the data and test it on another portion.

The cv argument is optional. When you leave it out, the function uses its default of three folds. When you pass a value, that value sets the number of folds, which is the subject of a later section.

Cross-validation evaluates a model by splitting the data into folds and then training and testing on different subsets. In K-fold cross-validation, the data is split into k subsets. The model is evaluated once for each fold, so the function produces one score per fold. Each fold takes a turn as the held-out test portion, and the remaining folds are used for training during that round.

The diagram shows the three-fold case, which is the default. Each fold is held out exactly once, and the model is trained on the remaining folds each time. The result is not one number for the whole procedure. It is a list of per-fold results, and that list is what the program prints.

The returned array is ordered by fold. The first element is the result from the round where the first fold was held out, the second element comes from the round where the second fold was held out, and so on. Because each value is the result of one held-out fold, the array length always matches the number of folds.

This mapping is the reason the array is more informative than a single average. If one element is much lower than the others, you know which held-out portion caused the drop, and you can investigate that part of the data rather than only seeing an overall number.

The cv parameter changes the number of folds. The rule for the output stays the same: the length of the returned array equals the number of folds. The next example keeps the same model and data and passes cv=5. Predict the length of the array before reading the answer.

Changing cv does not change what the individual numbers mean. It changes how many rounds of training and testing take place, which changes how many values appear in the output. If the length of the output does not match the cv value you passed, the cv argument is the first place to look.

When an output does not match your expectation, the useful question is where the divergence began. Rather than rereading the whole program, work through the chain in order. Each step narrows the search.

A grid search tries many parameter settings and records a cross-validation result for each one. It stores one row of results per setting, including the score from every split, along with the mean and standard deviation across those splits. When the search covers two parameters, the natural picture is a heat map: one axis holds one parameter, the other axis holds the second parameter, and each cell is colored by its mean score. The heat map turns a long table of numbers into a picture you can scan for patterns.

Light colors stand for high accuracy and dark colors stand for low accuracy. Each cell is one cross-validation result for one parameter setting. Reading the heat map therefore means asking where the light cells are, where the dark cells are, and how those regions are arranged relative to the two axes.

Many learners see a uniform heat map and assume the whole experiment is broken, so they throw it away and start over with a different model. That reaction is often premature. A uniform picture is a clue, and it points to a small set of specific causes that you can check one at a time. A uniform heat map can signal poorly chosen ranges or scaling, or a parameter with little effect, so it is a prompt to investigate rather than proof of failure.

The patterns of color are the main evidence. A constant color across the whole grid suggests the search ranges or scaling need attention. Color that changes across columns but stays constant down each column suggests that one parameter drives the score while the other has little effect. Color that changes along both axes suggests that both parameters matter. Each pattern points toward a different next step.

The source compares three heat maps produced from deliberately misspecified search grids. Each has a different arrangement of colors, and each arrangement points to a different diagnosis. Naming the pattern is the first step toward deciding what to change. The same model and data can give very different pictures when the ranges change, which is why the spacing of the values matters as much as the choice of model.

Adjusting the parameter grid after looking at cross-validation results is a legitimate and useful habit. The heat map guides those adjustments: it shows where accuracy changes, where it does not, and where the best region is pressing against the edge of the search. The adjustments should be made using cross-validation scores on the training data, which is where the heat map comes from. The final test set is kept out of this loop so that it still gives an honest estimate at the end.