Cross-Validation Fundamentals

When you tune a model with GridSearchCV, every candidate combination of parameters has to be evaluated on several different train-and-validate partitions of the data. That repeated evaluation is what makes the search trustworthy, but it also multiplies the amount of work. A grid with many candidates, run on a large dataset or with a model that takes a long time to train, can become expensive quickly.

This article looks at the cv parameter, which controls how those partitions are created, and at the option of replacing many partitions with one carefully chosen split. Before reading on, try to answer the question below from memory.

With k-fold cross-validation, the data is divided into k parts, usually called folds. In each iteration one fold is held out as the validation set and the remaining folds are used for training. Each candidate parameter set is therefore trained and checked several times, once for every fold taking its turn as the validation set.

Notice that the number of trainings grows with k. A candidate evaluated with 4-fold cross-validation is trained four times, and a grid of 12 candidates therefore requires 48 training runs. This is the cost that a single split later tries to reduce.

You do not have to specify anything to get cross-validation inside GridSearchCV. The defaults follow the same logic as the cross_val_score function. The choice depends on the kind of problem the estimator solves. A classifier predicts categories, so GridSearchCV uses stratified k-fold, which keeps the class proportions roughly the same in each part. A regressor predicts continuous values, so GridSearchCV uses plain k-fold instead. Stratification is a classification concern because class labels are what needs to be balanced across the parts.

The cv argument accepts more than a number of folds. You can pass any cross-validation splitter object, which is a tool that knows how to generate train and validation index sets from the data. Once you hand a splitter to GridSearchCV through cv, the search uses that splitter's logic in place of the default. An integer asks for that many folds using the default strategy for the estimator type, while a splitter object supplies the exact partitioning logic you want.

Ordinary cross-validation gives you several partitions, so each candidate parameter set is trained and checked several times. A different option is to ask for only a single split into one training set and one validation set. You get this by using ShuffleSplit or StratifiedShuffleSplit with n_iter=1. In recent releases the same setting is named n_splits. ShuffleSplit produces random train and validation partitions.

Before working through the numbers, consider the flow of data. The full dataset is divided once. Each candidate parameter set is then trained on the training portion and scored on the validation portion. Because there is only one partition, no candidate is judged on a second or third validation set.

The source gives a specific motivation for the single-split option: it might be helpful for very large datasets, or for very slow models. In both cases the cost of repeating training across many folds can dominate the total time. A single split keeps the search affordable while still giving each candidate a fair held-out evaluation.

Consider a grid of 12 candidates with a model that takes hours to train on the full dataset. With 5-fold cross-validation, the search would require 60 training runs. A single split reduces that to 12 training runs. The saving is real, but the score for each candidate now rests on one validation portion, so it is less stable than an average over several folds.