Generalizing the Learning Model
True error evaluates a prediction rule h with respect to labeled points drawn according to the probability distribution D.
From Training Performance to Generalization
A prediction rule h can be evaluated in two related ways. One view asks how likely h is to make an error on labeled points drawn according to the probability distribution D. This is the true error, also called the risk. The other view checks errors on the training data S. This is empirical risk. The distinction is central because the learner wants a rule with low true error, but does not know D and can directly access only S.
The learner's goal and the learner's available evidence are not the same thing: the goal concerns true error, while the directly available training data supports empirical-risk evaluation.
Following D to Labeled Points
D is the probability distribution associated with the labeled points used to evaluate h. True error considers how likely h is to make an error on labeled points drawn according to D. Therefore, true error is not limited to the particular examples already placed in S; it concerns the error behavior of h with respect to points associated with the distribution.
Tracing the Two Evaluation Sources
| Measure | Evaluation source | Role for the learner |
|---|---|---|
| True error, or risk | Labeled points drawn according to D | The error measure the learner wants to minimize |
| Empirical risk | Training data S | The error measure the learner can evaluate from available data |
The prediction rule h is the common object in both evaluations. True error uses labeled points associated with the unknown distribution D. Empirical risk uses the training data S that the learner has. The difference is not which prediction rule is being judged; it is where the evaluation points come from.
A Worked Evaluation Scenario
One Rule, Two Error Questions
Consider a prediction rule h, a probability distribution D, and a training set S. Identify which evaluation source is used by each question.
Question about future distribution-based behavior: Ask how likely h is to make an error on labeled points drawn according to D. This is a question about true error, or risk.
Question about the available examples: Check the errors made by h on the training data S. This is a question about empirical risk.
Compare the information available: The learner has access to S but does not know D. Therefore, the learner can directly evaluate empirical risk but cannot directly use knowledge of D to evaluate true error.
True error uses D as its evaluation source; empirical risk uses S. The learner wants low true error but can directly work with the training data S.
Mistakes About What Can Be Minimized
Saying that the learner directly minimizes true error using D.
The learner does not know the probability distribution D.
Fix:
State that the learner wants a prediction rule with low true error, while the learner has direct access to the training data S and can use it to measure empirical risk.Calling error on the training data true error.
Errors measured on S are the basis of empirical risk, not the distribution-based evaluation called true error.
Fix:
Use empirical risk when the evaluation source is S, and true error or risk when the evaluation source is labeled points drawn according to D.Ignoring the prediction rule h when comparing the two measures.
Both measures evaluate the same kind of object: a prediction rule h.
Fix:
Say that both measures evaluate h and differ in the source used for that evaluation.
Check the Evaluation Path
A learner has a prediction rule h and access to a training set S, but does not know D. For each statement, identify whether it describes true error or empirical risk: measuring errors on S; asking how likely h is to make an error on labeled points drawn according to D; choosing a goal for the quality of h; using information the learner can directly access.
Hints
- Look first for the evaluation source: D or S.
- Separate the learner's desired goal from the quantity the learner can directly evaluate.
- True error, or risk, asks how likely a prediction rule h is to make an error on labeled points drawn according to D. Empirical risk measures the error of h using the training data S. The learner wants low true error but does not know D. Because the learner does have access to S, empirical risk is the directly available evaluation source. The two measures concern the same prediction rule and differ in where their evaluation points come from.
Key Takeaways
- True error, or risk, evaluates a prediction rule h with respect to labeled points drawn according to D.
- Empirical risk evaluates the prediction error of h using the training data S.
- The learner wants a prediction rule with low true error.
- The learner does not know D but does have access to S.
- True error and empirical risk evaluate the same rule h through different evaluation sources.