TD Errors

From Rewards to Learning Signals

Reinforcement learning theory describes signals that help an agent learn from the consequences of its behavior. Neuroscientists have searched for brain activity that could play a comparable role. The reward prediction error hypothesis proposes that dopamine neuron activity represents one such signal: a reward prediction error. The hypothesis became influential because the behavior of TD errors in reinforcement learning theory shows striking parallels with the phasic activity of dopamine neurons.

A TD error is not simply a description of whether a reward occurred. It is a learning quantity that compares an existing estimate with a better estimate formed after new information arrives. This makes TD errors relevant to dopamine research: both are studied as signals that may indicate how learning should respond to what happens over time.

Dopamine Prediction Errors

The reward prediction error hypothesis links dopamine neuron activity with reward prediction errors. Its central evidence is the parallel between TD-error behavior and phasic dopamine activity. Phasic dopamine activity functions as a reinforcement signal that reaches multiple brain areas and supports learning.

Wolfram Schultz and colleagues conducted influential experiments in the late 1980s and 1990s. Those experiments revealed parallels between dopamine neuron activity and TD errors, helping lead scientists to the reward prediction error hypothesis. The important contribution was the comparison between a computational description and experimental observations, not the name of dopamine by itself.

The One-Step Comparison

δ_t = [R_t+1 + γV(S_t+1)] − V(S_t)

The expression has two sides to compare. V(S_t) is the current estimate for the state at time t. R_t+1 + γV(S_t+1) is a one-step-better estimate because it uses information from the next transition: the reward received and the estimated value of the next state. The TD error records the difference between that newer one-step estimate and the old estimate.

Why the Error Arrives Later

The label δ_t refers to the error in the estimate V(S_t), but calculating it requires R_t+1 and S_t+1. Those are next-step observations. Therefore, the TD error associated with time t becomes available at time t+1.

Building the Backup Target

After the transition from S_t to S_t+1, two pieces of new information are available: the next reward R_t+1 and the next state S_t+1. The reward is included directly. The estimate of the next state is multiplied by γ, the discount factor, and then included. Together they form the sample backup target R_t+1 + γV(S_t+1).

This target is called a sample backup target because it uses the particular reward and next state observed in one transition. The TD error then compares this target with the estimate that existed for S_t.

Three Learning Signals

The sign of the TD error describes how the one-step estimate compares with the current estimate. A positive error means the one-step estimate is greater than V(S_t); a negative error means it is lower; and a zero error means the two estimates match. As a reinforcement signal, this difference can indicate whether the current estimate should move upward, downward, or not change in response to the new transition.

This is why phasic dopamine activity is discussed as a possible reinforcement signal. The hypothesis connects changes in dopamine neuron activity with a computational difference that tells a learning system how new consequences compare with its prior prediction.

TD and Monte Carlo Errors

TD error and Monte Carlo error use different references. TD error uses a one-step sample backup: the next reward plus the discounted estimate of the next state. Monte Carlo error is based on the actual return. The source gives an important relationship between them: the Monte Carlo error can be written as a sum of TD errors.

Monte Carlo error = sum of TD errors

Common Interpretation Mistakes

• Treating a TD error as the reward itself

  The TD error compares a current estimate with a one-step estimate that includes both the next reward and the discounted next-state estimate.

  Fix: Use δ_t = [R_t+1 + γV(S_t+1)] − V(S_t).

• Assuming δ_t is available before the next transition

  The calculation requires R_t+1 and S_t+1, which are next-step observations.

  Fix: Associate δ_t with the estimate at time t, but recognize that it becomes available at time t+1.

• Confusing the TD target with the TD error

  That expression is the one-step estimate or sample backup target. The error is its difference from V(S_t).

  Fix: First form the target, then subtract the current estimate.

• Treating Monte Carlo error and TD error as the same comparison

  TD error uses a one-step sample backup, whereas Monte Carlo error is based on the actual return.

  Fix: Remember that the Monte Carlo error can be represented as a sum of TD errors.

Check Your Understanding

Key Takeaways

1. The reward prediction error hypothesis proposes that dopamine neuron activity represents a reward prediction error.
2. TD errors compare the current estimate V(S_t) with the one-step estimate R_t+1 + γV(S_t+1).
3. The error labeled δ_t becomes available at time t+1 because the next reward and next state are required.
4. Phasic dopamine activity is studied as a possible reinforcement signal because its activity shows parallels with TD-error behavior.
5. Monte Carlo error uses the actual return and can be written as a sum of TD errors.