Using scikit-learn for Model Selection
In scikit-learn, leave-one-out is provided by the LeaveOneOut class in the model_selection module.
Try it: K-Fold Cross-Validation and Model Selection
How hold-out and k-fold cross-validation choose among candidate models: each fold takes a turn as the validation set, the fold errors are averaged, the lowest average picks the model, and only then is the untouched test set used once — while tuning on the test set would report an optimistic number.
How it works
- Set 10 test points aside; they take no part in any choice.
- Optionally shuffle the 20 development points (seeded), then cut them into k folds (hold-out: one chunk validates, the rest trains).
- For every round, train each candidate polynomial on the other folds and measure its validation MSE on the held-out fold.
- Average each candidate's fold errors (with repeated shuffling: average the p run scores; p × k fits per candidate).
- Select the candidate with the lowest average, retrain it on all development points, and evaluate it once on the test set.
- For contrast, show which candidate the test set itself would have picked — that figure is leaked and optimistic.
Default run (12 steps): 20 development points (sorted by x as they arrived) and 10 test points set aside. Candidates: polynomial degree 0, 1, 2, 3, 4, 5, 6. Plan: 5-fold cross-validation, shuffled with seed 1 → 5 fits per candidate (p × k = 1 × 5), 35 in total. … 5-fold cross-validation picks degree 3 (score 0.0768); its honest test MSE is 0.0889. Picking by test error would report degree 4 at 0.0868 — a leaked, optimistic figure.
Simplified: One fixed toy dataset (20 development + 10 test points from sin(2πx) with noise) and polynomial least squares of degree 0–6 as the candidate models. Shuffling uses a seeded mulberry32 Fisher–Yates; folds are contiguous chunks like scikit-learn's KFold. Hold-out uses the first of k chunks as the validation set.
Loading the simulation…
Predicting the Output of a Leave-One-Out Run
When you evaluate a model, you need a test set that the model did not learn from. A single train-test split can be misleading, because the score depends on which rows happened to land in the test set. Cross-validation repeats the evaluation several times so that the performance estimate does not depend on one lucky or unlucky split of the data. Leave-one-out cross-validation is one of the frequently used methods in this family, and its defining move is simple: instead of holding out a chunk of the data for testing, it holds out exactly one sample at a time.
This article starts from code. The program below creates a LeaveOneOut splitter, passes it to cross_val_score together with a logistic regression model and the iris dataset, and then prints how many cross-validation iterations ran and the mean of their scores. Before you read on, make a prediction about the first printed line, because the number of iterations is the easiest place to notice when a leave-one-out result is not what you expected.
iris.dataholds the 150 feature rows and iris.target holds the 150 labels- loo is the splitter: it describes how the data is divided and does not fit a model
- cv=loo is the only argument that sets the strategy; the model and data are passed unchanged
len(scores)counts how many iterations produced a score, which is the first line to check
What do you think happens?
Before running the code, how many iterations will the first printed line report for the 150-sample iris dataset?
Reveal answer
150
Leave-one-out creates one split per sample. The iris dataset has 150 samples, so the run performs 150 iterations, and each iteration produces one score.
Holding Out One Sample at a Time
In k-fold cross-validation, the data is divided into k folds, and each fold takes a turn as the test set while the remaining folds train the model. Leave-one-out uses the same rotation but changes the size of each fold. If the dataset has $n$ samples, each fold contains exactly one sample, so the rotation runs once for every sample. Each test set is one point, and each training set contains the other $n-1$ points.
Because every sample is tested exactly once, the leave-one-out scores together cover the whole dataset. No sample is ever wasted as a permanent holdout, and every observation contributes both to training and to testing across the run. This is why the method is attractive when data is scarce. The diagram below shows a small dataset of five samples, where each iteration holds out a different single sample.
Each row of the diagram is a complete cross-validation iteration. Iteration 1 trains on samples 2 through 5 and tests on sample 1. Iteration 2 trains on samples 1, 3, 4, and 5 and tests on sample 2, and so on. Every sample appears as the single test point exactly once, which is the property that the definition depends on.
How LeaveOneOut Relates to k-Fold
Leave-one-out is k-fold cross-validation with the fold count set so that each fold is a single sample. In terms of the earlier notation, it corresponds to choosing $k = n$, where $n$ is the number of samples. A five-fold run on 150 samples would have 5 folds of about 30 samples each, with 5 iterations in total. Leave-one-out on the same data has 150 folds of one sample each, with 150 iterations.
The rotation itself is the same in both methods. What changes is the size of each test set and therefore the number of iterations. Because the number of iterations in leave-one-out follows the size of the data rather than a setting you choose, the count is a useful check: if you expected leave-one-out and the run reports a different count, the splitter or the data is not what you think.
Wiring LeaveOneOut into cross_val_score
In scikit-learn, leave-one-out is provided by the LeaveOneOut class in the model_selection module. You create the splitter object first and then pass it to a cross-validation function through the cv argument. The splitter itself does not fit a model. It only describes how the data should be divided, and cross_val_score performs the fitting and scoring for each split.
Read the pipeline as a loop. The splitter yields one split at a time. For each split, cross_val_score fits a fresh model on the training rows, scores it on the single held-out sample, and collects that score. After the last split, the collected scores are averaged, which is the value the program prints as the mean. Replacing loo with a k-fold splitter would change the number of iterations and the size of each test set, while the rest of the call would stay the same.
Tracing the Iteration Count on a Six-Sample Dataset
Predicting iterations and training sizes for six samples
A dataset has 6 samples. How many leave-one-out iterations run, and how many samples train each model?
- 1Identify the number of samples
The dataset has $n = 6$ samples.
- 2Apply the one-sample-per-fold rule
Each fold contains exactly one sample, so there are 6 folds and therefore 6 iterations.
- 3Count the training rows per iteration
Each training set contains the other $n-1 = 5$ samples.
- 4Check coverage
Across the 6 iterations, each sample is the single test point exactly once, so the 6 scores together cover the whole dataset.
The run performs 6 iterations, each training on 5 samples and testing on 1. If the printed count is not 6, the splitter or the data passed in is not what you expected.
The same reasoning applies to the iris dataset in the opening program. With 150 samples, the run performs 150 iterations, and each model is trained on 149 rows. The count you print is therefore a direct test of your expectation about the data size.
Mistakes When Reading a Leave-One-Out Result
Common mistake Expecting a small fixed number of scores from LeaveOneOut
What it looks likePrinting len(scores) and expecting 5 because the code was written with a five-fold habit.
Why it failsLeave-one-out does not use a chosen fold count. It produces one split per sample, so the count follows the size of the data.
Do this insteadCompare the printed count with the number of samples. For 150 samples, the expected count is 150.
Common mistake Assuming the iteration count reveals the model's accuracy
What it looks likeReading 150 and concluding that the model is good.
Why it failsThe iteration count only confirms that the splitting step produced the expected number of splits. It says nothing about the scores.
Do this insteadCheck the count first to rule out a splitting problem, then examine the scores separately.
Common mistake Treating the splitter as if it fits the model
What it looks likeExpecting loo to train logistic regression when the splitter is created.
Why it failsThe splitter only describes how the data should be divided. Fitting happens inside
cross_val_score.Do this insteadPass the model, the data, and the splitter to cross_val_score together, with the splitter as the cv argument.
Common mistake Forgetting that the mean is taken over the per-iteration scores, not over the original rows as a single fit
The Time Cost on Large Datasets
The main cost of leave-one-out is time. Because it trains one model per sample, the total work grows with the size of the dataset. A k-fold run trains only k models, so for a large dataset with $n$ samples and $k$ much smaller than $n$, leave-one-out requires far more model fits. The source notes that leave-one-out can be very time-consuming for large datasets.
The same structure gives leave-one-out an advantage on small datasets. Each model is trained on nearly all of the available data, since only one sample is removed. This can sometimes provide better estimates when data is scarce, because no observation is lost to a large permanent holdout. The trade-off is therefore clear: on small data, the extra fits are affordable and the estimate is valuable; on large data, the same method can become impractically slow.
Checking Your Understanding
A colleague runs leave-one-out on a dataset they believe has 12 samples, and the first printed line says 8. Name two things you would check, and explain what each check would tell you.
Hints
- Compare the printed count with the number of samples the dataset actually contains.
- Consider whether the data passed to
cross_val_scoreis the data the colleague intended, and whether the cv argument is really a LeaveOneOut object. - A mismatch in the count points to the splitting step or the data, before any question about model quality.
Key Takeaways
- 1Leave-one-out cross-validation is k-fold cross-validation where each fold is a single sample, so each sample is the test set exactly once.
- 2In scikit-learn, create a LeaveOneOut object and pass it as the cv argument to
cross_val_score. The number of iterations equals the number of samples. - 3When a result diverges, check the iteration count first, then the data held out in each split.
- 4Leave-one-out fits one model per sample, which can be very time-consuming on large datasets, but it can give better estimates on small datasets.
At a glance
- Leave-one-out cross-validation is k-fold cross-validation where each fold is a single sample, so each sample is the test set exactly once.
- In scikit-learn, create a LeaveOneOut object and pass it as the cv argument to
cross_val_score; the number of iterations equals the number of samples. - When a leave-one-out result diverges, check the iteration count first, since it reveals whether the splitter matches the data size you expected.
- Leave-one-out can be very time-consuming for large datasets, but it sometimes gives better estimates on small datasets.