Training and Validation Sets
Model selection uses validation to choose one predictor from several trained candidates.
The Selection Problem
Training does not always produce one unavoidable model. Starting with the same training set, you may use different algorithms or use one algorithm with different parameter settings. These choices can produce several candidate predictors. Model selection begins after those candidates exist: the task is to choose one of them. A fresh validation set provides a separate basis for making that choice.
The central workflow is: use the training set to create candidate predictors, use a fresh validation set to compare their errors, and select the candidate with the smallest validation error.
One Training Set, Several Candidates
The training set is used to produce a collection of candidate predictors. The candidates can arise because different learning algorithms are tried, or because one algorithm is run with different parameter choices. In either case, the common training set is the input to the procedures that produce the candidates. The result is not yet the final selected predictor; it is a collection from which one predictor can later be chosen.
Creating a Candidate Collection
Suppose the same training set is supplied to two different algorithms and to one algorithm with a different parameter setting. What exists before validation is used?
Use the common training set: The training set is supplied to each learning procedure or configuration.
Produce candidates: The procedures produce Predictor A, Predictor B, and Predictor C.
Prepare for selection: The candidate collection now exists, but no single candidate has been selected yet.
Model selection starts with several already-produced candidate predictors.
Keeping Validation Separate
The candidate collection depends on the training set because the training set is used to create the candidates. The validation set is fresh and independent of that collection. This separation gives model selection a separate basis for comparing the candidates rather than using the same data that produced the collection.
Comparing Validation Errors
Once the candidate collection has been formed, each candidate predictor is evaluated on the fresh validation set. The resulting validation errors are compared. Empirical risk minimization over the validation set means selecting the candidate with the smallest validation error. In this context, ERM is therefore a model-selection rule: it chooses one predictor from the candidates rather than creating the original candidate collection.
Selecting by Smallest Validation Error
Three candidate predictors are evaluated on the same fresh validation set. Predictor A has validation error 8, Predictor B has validation error 5, and Predictor C has validation error 11. Which predictor does validation-based ERM select?
Evaluate every candidate: The validation set is used to obtain an error for Predictor A, Predictor B, and Predictor C.
Compare the errors: The values are 8, 5, and 11. The smallest value is 5.
Apply the selection rule: ERM over the validation set retains the candidate with the smallest validation error.
Predictor B is selected because its validation error is the smallest of the three.
Training Versus Validation
| Training set | Validation set |
|---|---|
| Used to create the candidate predictors | Used to compare already-produced candidates |
| The candidate collection depends on it | The fresh set is independent of the candidate collection |
| Feeds different algorithms or parameter settings | Provides errors used for selection |
Treating training as if it must produce only one model
Different algorithms or different parameter settings can produce several candidate predictors from the same training set.
Fix:
Recognize that model selection begins after a candidate collection has been produced.Using the validation set to describe how the candidates were created
The validation set is used after the candidates exist, to provide a separate basis for comparing them.
Fix:
Use the training set to create candidates, then use a fresh validation set to evaluate and select among them.Selecting a candidate without comparing validation errors
Validation-based ERM selects the candidate with the smallest validation error.
Fix:
Evaluate every candidate, compare the resulting validation errors, and retain the candidate with the smallest one.Confusing the candidate collection with the selected predictor
The collection contains several candidates; model selection retains one of them.
Fix:
Keep the stages separate: collection first, selection second.
Practice the Workflow
A training set is used with three learning procedures, producing three candidate predictors. A fresh validation set is then used to evaluate all three. Candidate X has the smallest validation error, Candidate Y has a larger validation error, and Candidate Z has the largest validation error. Describe the complete model-selection process and identify which candidate is retained.
Hints
- Start with the role of the training set.
- Then explain what the validation set measures.
- Apply the smallest-validation-error rule.
Practice Answer
Explain the workflow for Candidates X, Y, and Z when Candidate X has the smallest validation error.
Create the collection: Use the training set with the learning procedures to produce Candidates X, Y, and Z.
Evaluate independently: Use the fresh validation set to evaluate all three already-produced candidates.
Select by ERM: Compare the validation errors and choose the candidate with the smallest one.
Candidate X is retained because validation-based ERM selects the candidate with the smallest validation error.
Key Takeaways
- One common training set can produce several candidate predictors through different algorithms or parameter settings.
- The validation set is used after the candidate collection has been formed.
- Validation-based empirical risk minimization selects the candidate with the smallest validation error.
- The candidate collection depends on the training set, while the fresh validation set is independent of that collection.
- Model selection is the process of choosing one predictor from the already-produced candidates.
Key Takeaways
- Training can produce several candidate predictors from one common training set.
- A fresh validation set compares the candidates after they have been produced.
- ERM over the validation set means selecting the candidate with the smallest validation error.
- The validation set is independent of the hypothesis collection produced from the training set.