Train-Test Split and Generalization
The test set is described as containing data that is new to the model.
Judging a Model Against Its Own Examples
Suppose you have a dataset and you want to build a machine learning model from it. A natural question follows immediately: how will you know whether the model is any good? The most tempting answer is to judge the model on the same data it was built from. That feels reasonable, but it measures something narrower than what you actually care about. It tells you how well the model fits the examples it has already seen, not how well it will handle new cases.
What do you think happens?
A model scores very high when it is evaluated on the same examples it was built from. Does that score tell you how it will perform on new, unseen examples?
Reveal answer
No, that score measures how well the model fits data it has already seen, not how well it handles new cases.
A model evaluated only on the data it was trained on tells you about fit, not about generalization to new data. The train-test split exists to keep these two questions apart.
Two Sets With Two Jobs
A train-test split divides a dataset into a training set and a test set. The training set is the material the model is built from. The test set is held back and used afterward to check the model's performance on data it was not built from.
The test set is described as containing data that is new to the model. That is exactly what makes it useful for judging generalization. The reason for splitting the data at all is to measure how well a model generalizes to new, previously unseen data. A model that fits its training set very closely may still make poor predictions on data it has never observed, so the training set score is not the quantity we care about. What matters is how well the model predicts for data that was not used during training.
The flow of data has a clear order. The full dataset is divided first. One portion feeds the model-building step, and the other portion is kept aside until the model has been built. Only then is that held-back portion used for evaluation. The diagram below shows the movement from the full dataset, through the split, to the two sets and the jobs they perform.
Notice that the test set reaches the evaluation step but never feeds into building the model. That separation is the whole point of the technique. If the test set were also used to build the model, it would no longer be data the model had not seen, and the evaluation would lose its meaning.
Tracing a Train-Test Evaluation in Python
A complete evaluation pipeline has three distinct moves, and each one has a specific job. First, the dataset is divided into a training set and a test set by train_test_split. Second, the fit method builds a model using only the training set. Third, the score method measures how the fitted model performs on the test set. The single number you see at the end depends on all three moves, but only the third one touches the test data.
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
X_train, X_test, y_train, y_test = train_test_split(X, y)
model = LogisticRegression()
model.fit(X_train, y_train)
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)- 1Line 1 and 2 import the split function and a logistic regression classifier.
- 2Line 4 divides the dataset. The four returned names keep the features and labels for each side separate.
- 3Line 6 creates an unfitted model object.
- 4Line 7 calls fit with X_train and
y_train. Only training rows touch the model here. - 5Line 9 scores the model on training rows. It shows fit, not generalization.
- 6Line 10 scores the model on test rows. This is the number that answers how the model handles unseen data.
- The split is the first move: it returns the training pieces and the test pieces
- fit receives only the training pieces, so the model is built from them alone
- score on the training pieces reports fit, which is not the quantity we need
- score on the test pieces is the fair check of generalization
Notice which lines touch which data. The test set reaches the model only through the score call on the final line. The model never adjusts itself using those rows. That separation is what makes the test score a fair check.
One Split, One Estimate
A single train-test split gives one estimate of generalization performance, and that estimate depends on which rows happened to land in the test set. Two different splits of the same dataset can produce two different test scores, even when the model and its settings are unchanged. One split therefore gives you one sample of how the model might behave, not a settled verdict.
The diagram uses row numbers only to show that the two splits assign different rows to the test set. The lesson is not about these particular numbers. It is that the score you obtain is partly a product of which rows ended up held back. This is the limitation that cross-validation is introduced to address.
Training Score Versus Test Score
The two scores answer different questions. The training score tells you how well the model fits the rows it was built from. The test score tells you how the model performs on rows it has never been fitted to. A high training score combined with a much weaker test score is the signature of a model that has learned its training examples without generalizing to new ones.
Cross-Validation: Taking Turns as the Held-Out Data
Cross-validation is introduced as a more robust way to assess generalization performance than a single split. Instead of relying on one split, the data is divided into several parts, and each part takes a turn as the held-out evaluation data while the others are used for training. Every part gets a chance to play the test role, so the final picture does not depend on one lucky or unlucky choice of rows.
The loop below uses four parts for illustration. In each round, one part is held out for evaluation and the remaining three are used for training. As the rounds progress, the held-out part rotates, so each part is used for evaluation exactly once. The source introduces cross-validation at this level of motivation; the detailed mechanics of running it in code come later in the course.
Grid Search and Choosing Parameter Settings
Every supervised model has parameter settings that are chosen before fitting, and the choice affects how well the model generalizes. Grid search is presented as an effective method for adjusting these parameters in supervised models to reach the best generalization performance. The idea connects directly to the rest of this article: a parameter choice should be judged by how it performs on held-out data, not by how it performs on the data the model was fitted to.
Evaluation tells you whether a setting works, and parameter selection uses that evaluation to pick a setting. If you tuned a setting by looking at the test score and then reported that same test score as your final result, the test set would have influenced the choice and would no longer give a fair check. Keeping the held-out data out of the fitting process is what keeps the final evaluation honest.
Beyond Accuracy and R Squared
The score method provides default measures. For classification it reports the fraction of correctly classified samples, which is accuracy. For regression the default measure is R squared. These defaults are a convenient starting point, but they are not the complete verdict on a model. A single number can hide differences that matter for a particular problem. Two classifiers with the same accuracy may behave very differently, so the choice of evaluation measure should match what you need the model to do.
Mistakes That Break the Split
Common mistake Judging the model on the data it was built from
What it looks likeCalling fit on all rows and then reporting the score on those same rows as the model's quality.
Why it failsThat measures how well the model fits known examples, not how it handles new data.
Do this insteadHold back a test set before fitting and score the model only on those held-back rows.
Common mistake Letting the test set influence model building
What it looks likeLooking at the test score, changing settings, and repeating until the test score looks good.
Why it failsThe test set was meant to contain data new to the model. Once it shapes the choices, it stops being a fair check.
Do this insteadUse the training data, or a separate held-out process, for choices, and reserve the test set for the final evaluation.
Common mistake Treating a single split as the final answer
What it looks likeReporting one test score as though any other split of the same data would produce the same number.
Why it failsA single split gives one estimate, and that estimate depends on which rows landed in the test set.
Do this insteadUse cross-validation to get a more robust picture of generalization performance.
Common mistake Preparing the data using information from all rows before splitting
What it looks likeComputing a summary of the full dataset and using it to prepare the training and test rows alike, before the split is made.
Why it failsThis is a generated illustration of the same principle: if information from the test rows reaches the model-building step, the test rows are no longer new to it.
Do this insteadMake the split first, then let any preparation step learn only from the training rows.
Practice Reading a Split
Look again at the code from the earlier section. Name every line that touches the test rows, and explain what the resulting number tells you. Then say whether the training score could be used to claim the model generalizes well.
Hints
- Only the test rows should be used by the score call that reads the test score.
- Ask which rows the model was fitted on, and whether those same rows can show how it handles new data.
- A high training score shows fit to known rows, which is a different question from generalization.
Summary of the Evaluation Pipeline
- 1A model evaluated only on its own training data shows fit, not generalization to new data.
- 2A train-test split sends one portion of the data to fit to build the model, and holds the other portion back until score measures performance on it.
- 3A single split gives one estimate that depends on which rows landed in the test set, so cross-validation rotates the held-out part to get a more robust picture.
- 4Grid search adjusts parameter settings using held-out evaluation, and default measures such as accuracy and R squared are only part of the picture.
At a glance
- A model's score on its own training data measures fit, not how well it generalizes to new data.
- A train-test split builds the model from the training set and evaluates it on the held-back test set, which contains data new to the model.
- A single split gives one estimate that depends on which rows landed in the test set; cross-validation takes turns holding out each part for a more robust assessment.
- Grid search tunes parameter settings using held-out performance, and default measures such as accuracy and R squared are only part of the picture.