Train-Test Splitting in Python
A score computed on the training data describes how well the model fits examples it already saw, not how well it generalizes.
Why a High Training Score Misleads
A very common first instinct is to fit a model to a dataset, ask how well it did on that same dataset, and treat a high number as proof of quality. The reasoning feels natural: the model saw these examples, so it should get them right. The trouble is that a model is not built to remember its training examples. It is built to make predictions for data it has never seen, and a score computed on the training data does not measure that ability.
This chapter takes up the question of how good a supervised model actually is and how to choose its settings so that it works well in practice. Supervised methods, meaning regression and classification, are the focus here, because their predictions can be checked against known answers.
What do you think happens?
You train a classifier and compute its score twice: once on the examples it was trained on, and once on examples it has never seen. Should the two numbers match, and if one is more trustworthy, which one and why?
The Three-Step Evaluation Workflow
The evaluation workflow follows a fixed order of operations. First, the dataset is divided into a training set and a test set using the train_test_split function. Second, a model is built on the training set by calling the fit method. Third, the model is evaluated on the test set using the score method. For classification, score computes the fraction of correctly classified samples, so it is a single number that summarizes how often the model's predictions on held-out examples were right.
The reason for splitting the data is to measure generalization. Generalization means how well a model performs on new, previously unseen data. The interest is not in how closely the model reproduces the examples it was fitted to, but in how well it predicts examples that were not observed during training. The test set plays the role of that unseen data, so the score computed on it is the closest thing this workflow offers to an estimate of real-world performance.
Tracing Where Each Row Goes
Follow a single dataset through the split. Every row starts in the same full dataset. The train_test_split function sends some rows to the training set and holds the rest back as the test set. The training set is the only portion the model learns from through fit. The test set is not used to fit the model; it is kept aside and used afterward to measure the result.
Mixing these two roles is what produces the misconception described at the start. If rows that the model was fitted on also appear in the portion used for scoring, the score no longer tells you about examples the model has not seen.
Tracing the Fit and Score Calls
Now trace the order of the calls. The fit method runs first and receives the training rows. Only after that does score run, and it receives the test rows. Each call takes the inputs and labels from one portion, so the two calls never see the same data unless the split was done incorrectly.
Code: Splitting, Fitting, and Scoring
The following illustrative snippet follows the same workflow and computes the score twice so the two roles can be compared directly. It is an original example built on the pattern described above.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)
print('Training score:', model.score(X_train, y_train))
print('Test score:', model.score(X_test, y_test))- 1Lines 1 to 3 import the split function, a sample dataset, and a classifier.
- 2Line 5 loads features in X and labels in y.
- 3Line 6 splits both X and y into training and test portions.
- 4Lines 8 and 9 create the model and call fit on the training portion only.
- 5Line 11 scores the model on the training rows, which describes examples it already saw.
- 6Line 12 scores the model on the test rows, which is the estimate of generalization.
train_test_splitdivides the full dataset into training and test portions- fit builds the model using only the training rows
- The first score call uses the training rows the model already learned from
- The second score call uses the held-out test rows
Reading the Two Scores
The pair of scores tells a story about the model. A model that has essentially memorized its training examples will score high on the training rows and noticeably lower on the test rows. A model that generalizes will show training and test scores that are close to each other. The gap between the two numbers is therefore a practical signal of whether the model learned patterns that carry over to new data.
A single split gives one estimate, and that estimate depends on which examples happened to land in the test set. Keep this in mind when reading any single pair of numbers.
Worked Example: Interpreting a Result
Deciding Whether a Model Is Ready
A classifier is trained with the three-step workflow. Its score on the training rows is very high, and its score on the test rows is much lower. Should the training score be reported as the model's quality?
- 1Identify the purpose of the test score
The test rows were held back from fit, so the score on them estimates how the model handles unseen data.
- 2Identify the purpose of the training score
The training rows were used to build the model, so a high score there mainly shows the model fits examples it already saw.
- 3Compare the two
A large gap suggests the model is not generalizing well, even though the training number looks impressive.
- 4Choose the reported figure
The test score is the figure that reflects generalization, so it is the one to report as the model's quality.
The training score should not be reported as the model's quality. The test score is the estimate of performance on unseen data, and a large gap is a warning sign.
Common Mistakes With Splitting and Scoring
Common mistake Reporting the score computed on the training data as the model's quality
What it looks likeCalling score on X_train and presenting that number as the final result
Why it failsThe training rows were used to fit the model, so the number describes examples the model already learned from.
Do this insteadReport the score computed on the held-out test rows.
Common mistake Fitting the model on the test rows
Why it failsThe test set is supposed to stand in for unseen data. Once it is used for fitting, the score no longer estimates generalization.
Do this insteadCall fit only with the training portion produced by the split.
Common mistake Treating one split's score as a precise measure of performance without asking which examples landed in the test set
Next Steps: Cross-Validation and Grid Search
Two extensions follow from the single split. The first is cross-validation, which the source presents as a more robust way to assess generalization performance than one train-test split. The second is grid search, which the source describes as an effective method for adjusting the parameters of supervised models to achieve the best generalization performance.
Together these ideas connect evaluation to parameter selection. The same generalization question that guides how you read a score also guides which parameter settings you keep. The source also notes that the default score measures are accuracy for classification and $R^{2}$ for regression, and that other measures can be used for these tasks.
Practice: Predict the Pattern
Using the generated snippet as a model, imagine you change the code so that fit is called with all rows of X and y, and then you call score on the same rows. Predict what this does to the meaning of the second number, and explain whether it still estimates generalization.
Hints
- Ask which rows the model was fitted on.
- Ask whether those same rows are the ones the score is computed on.
- Compare the result to the workflow of splitting first and scoring only on the test set.
Summary of the Workflow
At a glance
- A score computed on the training data describes how well the model fits examples it already saw, not how well it generalizes.
- The workflow splits data with
train_test_split, builds the model with fit on the training set, and measures generalization with score on the held-out test set. - Comparing a training score with a test score reveals whether a model has memorized its training examples or learned patterns that carry over to new data.
- A single split gives one estimate that depends on which examples fall into the test set, so cross-validation offers a more robust assessment.
- Grid search adjusts model parameters to achieve the best generalization performance.