Eligibility Traces in Reinforcement Learning

The Delayed-Reward Problem

A reinforcement-learning system may need to connect two events separated in time: an earlier synaptic event and a later reinforcement signal. The central question is how the system can preserve enough information about the earlier event for the later reward to affect the appropriate synapse. Miller's 1981 Law-of-Effect-like proposal addressed this problem with synaptically local contingent eligibility traces.

A Synapse Becomes Eligible

In Miller's proposal, eligibility is local to a synapse. An earlier sensory or synaptic event does not immediately need to produce the final learning change. Instead, that event can leave the synapse temporarily eligible. If a reinforcement signal arrives afterward, the signal can be related to the synapse that was previously marked. The trace therefore addresses temporal credit assignment: it helps connect a later outcome with an earlier event.

Miller's Sensory Analyzer

Miller's proposal also included a sensory analyzer unit with a critic-like role. The important function is not merely sensing input. The analyzer supplied reinforcement signals to neurons, providing the signal that could guide changes at synapses that had become eligible. This anticipates a central feature later associated with actor-critic architectures: one part of the system contributes behavior, while a critic-like mechanism supplies information related to reinforcement.

The Hedonistic Synapse

Seung's hedonistic synapse gives the learning problem a synapse-level interpretation. The synapse changes its neurotransmitter release probability according to which event is followed by reward: release or failure to release. The relevant distinction is between synaptic activity and the probability of the synapse's future neurotransmitter release. A hedonistic synapse therefore expresses learning as a reward-dependent change in release probability, rather than treating every synaptic event as an identical outcome.

Historical Connections

The shared question across these ideas is how to connect a past event with a later reinforcement signal. The source connects Miller's proposal with the general features of reward-modulated STDP and connects the later actor-critic architecture with TD error. The useful comparison is therefore structural: an earlier synaptic or behavioral event is related to an eligibility-like record, a later reward-related signal arrives, and learning uses the relationship between the two. This relationship should not be mistaken for proof that all three approaches implement the same mechanism.

Bacterial chemotaxis belongs here as a metaphor for the problem of using consequences to guide future behavior. It should not be confused with the neural learning mechanisms themselves. The mechanisms discussed in this article are eligibility traces, reinforcement signals, reward-dependent synaptic release probability, reward-modulated STDP, and actor-critic use of TD error. The metaphor helps frame the learning question; it does not replace the variables and synaptic mechanisms used in those proposals.

Common Mistakes

• Treating an eligibility trace as the reinforcement signal itself.

  The trace marks an earlier synaptic event, while Miller's critic-like sensory analyzer provides reinforcement signals to neurons.

  Fix: Keep the roles distinct: eligibility identifies a synapse that can receive later credit, and the analyzer supplies the reinforcement-related signal.

• Assuming that a later reward automatically changes every synapse.

  Miller's rule is described as using synaptically local contingent eligibility traces.

  Fix: Ask which synapse was made eligible by the earlier event before assigning credit.

• Confusing release probability with a single release event.

  The hedonistic synapse changes its neurotransmitter release probability according to whether reward follows release or failure to release.

  Fix: Separate the event that occurred from the learned probability governing future release.

• Claiming that Miller's proposal and actor-critic methods are identical.

  The source presents Miller's proposal as anticipating an important actor-critic feature and connects later actor-critic methods with TD error.

  Fix: Describe them as historically and conceptually connected through the problem of linking past events to later reinforcement.

• Treating bacterial chemotaxis as the neural learning rule.

  The metaphor and the learning mechanisms are different levels of explanation.

  Fix: Use chemotaxis only to frame the consequence-guided learning problem, then return to the neural variables.

Check Your Understanding

The Mechanism in One Pass

1. An earlier sensory or synaptic event occurs.
2. A synaptically local contingent eligibility trace makes the relevant synapse temporarily eligible.
3. Miller's critic-like sensory analyzer supplies a reinforcement signal to neurons.
4. The later reinforcement signal can affect the synapse whose earlier activity left it eligible.
5. In the hedonistic-synapse proposal, reward changes neurotransmitter release probability according to whether reward follows release or failure to release.
6. Reward-modulated STDP and actor-critic methods, including the use of TD error, are connected by the same broad problem of assigning later reinforcement to earlier events.