Reward Prediction and Animal Learning

Prediction or Control

Animal learning can involve two different problems. In a control problem, the learner must act in a way that produces a reinforcing stimulus. In a prediction problem, the learner learns that one event signals another. The distinction between instrumental conditioning and classical conditioning follows from this difference: instrumental conditioning concerns behavior that produces reinforcement, whereas classical conditioning concerns predictive information about events.

Tracing Event A

TD Prediction Across Time

Temporal-difference learning provides a reinforcement-learning account of prediction that parallels classical conditioning. In this use of TD learning, the learner is not primarily selecting an action to produce reinforcement. Instead, it is learning predictive information about events. The important addition is temporal structure: the model considers events within an individual trial, so the learner's prediction can be related to the position of an event within that trial.

The timeline matters because TD learning treats a trial as containing successive moments or states rather than as one undivided event. A cue can therefore be considered in relation to what follows it within the same trial. This is why learning to predict with TD learning corresponds to the classical-conditioning pattern: the model tracks predictive relations between events, while its temporal structure represents where those events occur in the unfolding trial.

Rescorla-Wagner and TD

The TD model is described as a generalization of the influential Rescorla-Wagner model. The key extension is not a change from learning to prediction into learning to control. Both are relevant to learning about events. Instead, TD adds temporal structure by considering events within individual trials. This lets the model represent the sequence and timing of intermediate events rather than treating a trial as lacking internal timing.

Second-Order Conditioning

Second-order conditioning describes how a predictor of a reinforcing stimulus can itself become reinforcing. The important sequence is that a new cue precedes an already learned predictor. The new cue does not need to be directly paired with the reinforcing stimulus in order to acquire significance; its relationship to the established predictor can give it predictive value.

Common Misunderstandings

• Treating classical conditioning as behavior that produces reinforcement.

  Classical conditioning is defined by the independence of the reinforcing stimulus from the animal's behavior.

  Fix: Ask whether the learner is acquiring control through action or prediction about events.

• Describing TD learning in this context as primarily an action-selection account.

  The relevant reinforcement-learning parallel is TD learning used for prediction.

  Fix: Focus on how the learner predicts a later reinforcing stimulus from an earlier event.

• Treating the TD model and the Rescorla-Wagner model as identical.

  The TD model extends the Rescorla-Wagner account by adding temporal structure.

  Fix: Represent the events and intermediate moments within the trial.

• Assuming second-order conditioning requires direct pairing of every cue with the reinforcing stimulus.

  Second-order conditioning concerns a new cue that precedes an already learned predictor.

  Fix: Trace the predictive chain from the new cue to the learned predictor and then to the reinforcing stimulus.

Check Your Understanding

1. Classical conditioning is a prediction problem in which the reinforcing stimulus is independent of the animal's behavior. TD learning parallels this pattern when it is used to learn predictions about events rather than to select actions that produce reinforcement. The TD model extends the Rescorla-Wagner account by representing temporal structure within individual trials. Second-order conditioning follows when a new cue precedes an already learned predictor and thereby acquires predictive value itself.