TD Error Concept

A Changing Expectation

The TD error concept focuses on a change in an estimate. A system first has an old estimate of the reward expected in the future. It then forms a new estimate. The difference between the old and new estimates is treated as an error signal. The central idea is therefore not merely that a reward exists, but that the system's expected-reward estimate has changed.

Tracing the Two Estimates

Keep the sequence clear. First, there is an old estimate of expected future reward. Next, a new estimate becomes available. Finally, the two estimates are compared. The result of that comparison is the TD error. This sequence matters because the hypothesis is about an error between estimates, not simply about the presence or absence of a reward.

1. Start with the old estimate of the reward expected in the future.
2. Form a new estimate of the reward expected in the future.
3. Compare the new estimate with the old estimate.
4. Treat the difference between them as an error signal.
5. Relate that error signal to the proposed function of phasic dopamine-producing neuron activity.

A Numerical Difference

This example is intentionally simple. Its purpose is to separate the two quantities: 5 is the old estimate, 8 is the new estimate, and 3 is the difference between them. In a fuller reinforcement-learning treatment, the estimates and their update rules can be more elaborate, but the concept described here still begins with the difference between an old and a new expected-reward estimate.

Link to Phasic Dopamine

The TD error concept is connected to phasic activity in dopamine-producing neurons. The reward prediction error hypothesis proposes that one function of this phasic activity is to deliver the error signal to target areas throughout the brain. In this view, the activity is related to the difference between old and new estimates of expected future reward.

What the Hypothesis Does Not Say

• Treating the hypothesis as a claim about reward alone.

  The hypothesis begins with an old estimate and a new estimate of expected future reward. The important quantity is their difference.

  Fix: Ask what changed in the expected-reward estimate, then identify the difference between the two estimates.

• Confusing the old estimate with the new estimate.

  The error requires a comparison between two estimates that occupy different stages in the sequence.

  Fix: Label the starting value as the old estimate and the later value as the new estimate before comparing them.

• Saying that the hypothesis proves phasic dopamine activity is identical to reward.

  The account describes phasic dopamine-producing neuron activity as proposed to deliver an error signal to target areas. That is different from saying the activity is the reward itself.

  Fix: State the relationship narrowly: the activity is proposed to deliver the error associated with the change in expected reward.

• Turning a proposed function into an exhaustive explanation.

  The source says the TD error concept helps account for many features of phasic dopamine neuron activity and describes one proposed function. Those statements do not automatically cover every feature or function.

  Fix: Use qualified language such as proposed, helps account for, and one function.

Check Your Understanding

Key Takeaways

1. TD error is the difference between an old estimate and a new estimate of expected future reward.
2. The reward prediction error hypothesis focuses on a change in an estimate, not merely on the existence of a reward.
3. Phasic dopamine-producing neuron activity is proposed to deliver this error signal to target areas throughout the brain.
4. The TD error concept helps account for many features of phasic dopamine neuron activity, but the focused claim should not be expanded into an exhaustive explanation.
5. A simple numerical illustration can identify the error by comparing the two reward estimates.