Value Functions and Prediction Errors

When Outcomes Defy Expectations

Reinforcement learning does not consider only whether a reward occurs. It also considers how the actual reward compares with the reward that was expected. That comparison produces a reward prediction error, or RPE. An RPE is therefore a difference signal: it represents a mismatch between expectation and outcome rather than the reward itself.

Reward Versus Mismatch

The reward is the outcome that occurs. The reward prediction error is a separate signal about the relationship between that outcome and the expectation that preceded it. If the actual reward is better than expected, the mismatch is positive. If it matches the expectation, the mismatch is zero. If it is worse than expected, the mismatch is negative. These descriptions concern the comparison, not the reward's identity.

Predictions Across Time

A temporal-difference error, or TD error, is a temporal-difference form of error signal. The temporal part matters because the comparison is considered across successive moments: a prediction at one moment is related to what is encountered at that moment and to the value predicted at another moment. In this framework, an error is not merely a label for an unexpected reward; it is part of a sequence of predictions and comparisons over time.

The diagram is a conceptual time sequence, not a numerical learning rule. Its key point is that TD error is the temporal-difference form of an error signal. The source makes a more specific neuroscience connection: phasic activity of dopamine-producing neurons is described as conveying TD errors.

Dopamine as a Candidate Error Signal

The reward prediction error hypothesis connects a computational idea from reinforcement learning with an observed biological signal. Reinforcement-learning theory defines an error signal based on expected and actual rewards. Neuroscience investigates activity in dopamine-producing neurons. The hypothesis proposes that dopamine neuron activity signals reward prediction errors, and that phasic activity of dopamine-producing neurons conveys temporal-difference errors.

Value as Future Expectation

A value function belongs to the prediction side of this discussion: it represents the expected future reward associated with a situation or state. The important contrast is between a prediction and the error produced when the later outcome differs from that prediction. A value estimate is therefore not identical to the reward that eventually occurs, and neither one is identical to the error signal describing their mismatch.

This flow separates three roles. The situation is what the learner or model is evaluating. The value function supplies an expectation about future reward. The prediction error records how the eventual outcome compares with that expectation. Keeping these roles separate prevents the value prediction or reward from being mistaken for the error signal.

A Three-Outcome Trace

Reading Neural Results Carefully

1. Observe the neural activity.
2. Compare the activity with the definitions of possible computational signals.
3. Ask whether the activity resembles an RPE or a TD error.
4. State the dopamine connection as a proposed interpretation rather than treating the label as automatic.
5. Preserve the distinction between the reward, the expectation, and the mismatch signal.

• Calling the reward itself an RPE.

  The reward is 8; the RPE is the signal about the difference between 8 and 5.

  Fix: Name the outcome and the mismatch separately.

• Treating every dopamine response as automatically proving an RPE.

  The source says that neural activity must be compared with several possible interpretations.

  Fix: Describe dopamine as a proposed or candidate signal for RPEs, then explain why the observed pattern resembles that theoretical signal.

• Using TD error and RPE as if the terms had exactly the same scope.

  An RPE is the broad computational idea of a difference between expected and actual reward, while TD error is a type of error signal used in temporal-difference learning.

  Fix: State the broad RPE idea first, then identify TD error as the temporal-difference form.

• Ignoring the time dimension in TD error.

  The temporal-difference description concerns error signals across time.

  Fix: Explain which predictions and outcomes are being related across successive moments.

Check Your Understanding

A strong answer should say that the outcome of 10 matches the expectation, whereas the outcome of 6 is worse than expected and therefore produces a negative mismatch. It should also preserve the interpretive chain: neural activity is observed, compared with theoretical signal definitions, and then considered in relation to the hypothesis that dopamine signals RPEs and that phasic dopamine-producing-neuron activity conveys TD errors.

Key Takeaways

1. A reward prediction error is a difference signal based on the comparison between an expected reward and an actual reward.
2. The reward is the outcome; the RPE is the signal about the mismatch between the outcome and the expectation.
3. A temporal-difference error is a temporal-difference form of error signal.
4. The reward prediction error hypothesis proposes that dopamine neuron activity signals RPEs.
5. Phasic activity of dopamine-producing neurons is described as conveying TD errors, but neural activity should be compared with alternative interpretations before it is labeled an RPE.