Correct Labeling Function
Classifier error measures the probability of an incorrect prediction on a randomly drawn instance.
Why One Failed Prediction Is Not the Error
Suppose a classifier receives an instance and predicts a label. That one prediction is either correct or incorrect, but classifier error is not the name of one particular failure. Classifier error describes the probability that a randomly drawn instance will receive an incorrect prediction.
To reason about the error, imagine repeatedly drawing instances according to an underlying probability distribution. For every drawn instance, compare the classifier's prediction with the correct label. The overall error is the probability of landing in the incorrect-prediction outcome.
What do you think happens?
If a classifier gives the correct label for one randomly drawn instance, is that single outcome itself a classifier error?
Reveal answer
Answer: No, because the prediction agrees with the correct label.
A single trial is either correct or incorrect. Classifier error is the probability of the incorrect outcome across randomly drawn instances, not the name of every individual trial.
From a Drawn Instance to an Error
Three ingredients determine whether a prediction counts as an error. First, the distribution D determines which instances are likely to be drawn. Second, the correct labeling function f gives the correct label f(x) for an instance x. Third, the classifier h produces its prediction h(x). An error occurs when h(x) and f(x) disagree.
Correct Labels and Classifier Predictions
For a particular instance x, the correct labeling function supplies f(x), while the classifier supplies h(x). These labels play different roles. The function f represents the correct label used for comparison; h represents the classifier's attempt to produce that label. Agreement means the particular prediction is correct. Disagreement means that particular prediction is incorrect.
This comparison also explains why the correct labeling function is essential. Without f(x), there is no reference label against which to decide whether h(x) is correct. The error therefore depends on both what the classifier predicts and what the correct labels are for the instances that the distribution can produce.
Counting Errors in an Equal-Likelihood Collection
Consider a generated collection of four instances. Assume the distribution gives each instance equal likelihood. For each instance, compare f(x), the correct label, with h(x), the classifier's prediction.
| Instance | Correct label f(x) | Prediction h(x) | Outcome |
|---|---|---|---|
| A | red | red | correct |
| B | blue | red | incorrect |
| C | green | green | correct |
| D | blue | green | incorrect |
Generated equal-likelihood collection for calculating classifier error.
Classifier Error for Four Equal-Likelihood Instances
Determine the classifier's error from the generated collection.
Compare labels: Instance A is correct because h(x) and f(x) are both red. Instance B is incorrect because the prediction red differs from the correct label blue. Instance C is correct because both labels are green. Instance D is incorrect because the prediction green differs from the correct label blue.
Identify incorrect cases: The incorrect-prediction cases are B and D, so there are two incorrect cases among four total cases.
Use equal likelihood: Because all four instances are equally likely in this generated collection, the probability of an incorrect prediction is the fraction represented by those two cases.
The classifier's error is 2 out of 4, or 1/2, which is 50 percent.
Three Names for One Quantity
In this framework, generalization error, risk, and true error are names for the same distribution-based quantity: the probability that h(x) disagrees with f(x) when x is drawn according to D.
| Term | Meaning in this framework |
|---|---|
| Generalization error | The probability of an incorrect prediction on an instance drawn from the distribution. |
| Risk | The same distribution-based probability of disagreement between h(x) and f(x). |
| True error | The same distribution-based probability of disagreement between h(x) and f(x). |
Mistakes in Error Calculations
Treating one incorrect prediction as the entire classifier error.
That observation identifies one incorrect outcome, not the probability of incorrect outcomes over randomly drawn instances.
Fix:
Consider the distribution of possible instances and determine the probability assigned to all cases where h(x) disagrees with f(x).Comparing the prediction with something other than the correct label.
The relevant comparison is h(x) versus f(x), the classifier's prediction versus the correct labeling function's label.
Fix:
For each instance, compare h(x) directly with f(x).Using the raw number of incorrect instances when likelihoods are unequal.
The distribution determines how likely each instance is to be drawn. Equal counts do not necessarily represent equal probability.
Fix:
Determine how much probability the distribution assigns to the incorrect-prediction cases.
Practice the Procedure
A generated collection contains three equally likely instances. For instance P, h(x) agrees with f(x). For instance Q, h(x) disagrees with f(x). For instance R, h(x) agrees with f(x). What is the classifier's error for this collection?
Hints
- Count the instances where h(x) and f(x) disagree.
- Because the instances are equally likely, express the error as the number of incorrect cases out of the total number of cases.
The answer is one incorrect case out of three, so the error is 1/3. The same procedure changes when the distribution is not uniform: instead of relying on the raw fraction of cases, determine the probability assigned to the incorrect case or cases.
Key Takeaways
- Classifier error is the probability of an incorrect prediction for a randomly drawn instance.
- An individual prediction is evaluated by comparing h(x), the classifier's prediction, with f(x), the correct label.
- The distribution D matters because it determines how likely each instance is to be drawn.
- Generalization error, risk, and true error refer to the same distribution-based error quantity in this framework.
- For equally likely instances, error can be calculated as the proportion of instances on which h(x) and f(x) disagree; with unequal likelihoods, use the probability assigned to the incorrect cases.
Key Takeaways
- Classifier error measures the probability of an incorrect prediction, not one isolated failure.
- The comparison is between h(x) and f(x): the classifier's prediction and the correct label.
- The data distribution determines how much each incorrect-prediction case contributes to the error.
- Generalization error, risk, and true error are equivalent names for this quantity in the stated framework.
- With equal likelihood, count disagreements and divide by the total; with unequal likelihood, add the probabilities of the disagreement cases.