Dopamine and Neuromodulatory Signals
Eligibility traces preserve synapse-specific information about earlier activity.
Why Timing Matters
A neuron may participate in an action before the consequence of that action is available as a learning signal. This creates a timing problem: the synapse is active first, while the reinforcing neuromodulatory signal may arrive later. Synaptic eligibility traces provide the connection between these two moments. They record earlier activity at a particular synapse so that a later reinforcing signal can use that record when synaptic efficacy is adjusted.
The essential sequence is activity at a particular synapse, an eligibility record at that synapse, and a later reinforcing signal that modifies the synapse's efficacy.
The Synapse-Specific Record
An eligibility trace is a record associated with a synapse rather than a single undifferentiated record for the whole neuron. When activity involves a particular synapse, that synapse becomes eligible for possible later modification. The trace preserves information about the synapse's earlier participation. It matters because a reinforcing signal does not have to arrive at exactly the same moment as the original synaptic activity.
Following Two Synapses Through Time
Suppose Synapse A and Synapse C participate in a neuron's activity, while Synapse B does not participate in that activity. A reinforcing neuromodulatory signal arrives later.
Record earlier participation: The activity involving Synapse A and Synapse C leaves those synapses eligible for possible modification. The record is attached to the participating synapses, not simply to the neuron as a whole.
Wait for the later signal: The reinforcing signal arrives after the earlier synaptic activity. The eligibility record connects the earlier event to this later learning signal.
Select eligible synapses: The later reinforcing input can modify the efficacy of Synapse A and Synapse C because they are eligible. Synapse B has no stated eligibility record in this example.
Eligibility traces preserve which synapses participated earlier, allowing a later reinforcing signal to act on those synapses.
Reinforcement Changes Efficacy
The reinforcing neuromodulatory signal does not need to recreate the earlier synaptic activity. Instead, it arrives while the relevant synapse is eligible and uses that stored activity information when synaptic efficacy is adjusted. In this way, eligibility is the temporary bridge between a synapse's participation and a later consequence.
In the learning-rule interpretation, an actor unit can participate in an action before the consequence of that action becomes available as a learning signal. Its eligibility trace preserves the earlier synaptic and actor activity until the reinforcing signal arrives, connecting the action with its later consequence.
Actor and Critic Traces
Actor and critic learning rules both use eligibility traces, but their traces carry different information. An actor trace is contingent: it depends on the actor unit's input and also on the actor unit's output. A critic trace is non-contingent: it does not involve the critic unit's output. The distinction reflects the different roles of the learning rules, not whether the synapse has an activity record.
| Trace | Contingency | Information included |
|---|---|---|
| Actor trace | Contingent | Actor input and actor output |
| Critic trace | Non-contingent | Does not include critic output |
Biological Interpretation
Klopf's hedonistic-neuron hypothesis supplies an intuition for why eligibility is useful. A neuron is treated as part of a closed loop: its activity can influence later input through feedback within the nervous system and body or through the external environment. The neuron therefore adjusts synaptic efficacies according to rewarding or punishing consequences associated with its action potentials. In this view, a synapse becomes eligible when it participated in the neuron's firing, and a reinforcing signal selects that eligible synapse for modification.
The interpretation becomes more specific when actor and critic learning rules are related to brain circuits. These learning rules correspond to plasticity rules for corticostriatal synapses. Such synapses carry signals from the cortex to principal neurons in dorsal and ventral subdivisions of the striatum, and they also receive signals from dopamine neurons. Eligibility traces provide the bridge between earlier activity at a corticostriatal synapse and later neuromodulatory input relevant to changing its efficacy.
Common Mistakes
Treating eligibility as if it were the same as synaptic modification.
Eligibility records earlier participation; the later reinforcing signal is what can use that record to modify efficacy.
Fix:
Separate the sequence: activity establishes eligibility, then later reinforcement can modify the eligible synapse.Treating the eligibility trace as one neuron-wide record.
The trace preserves synapse-specific information about earlier activity.
Fix:
Ask which particular synapses participated and became eligible.Calling the critic trace contingent because it belongs to a learning unit.
The critic trace is non-contingent and does not involve the critic unit's output.
Fix:
Remember that actor traces include actor input and output, while critic traces do not include critic output.Assuming the reinforcing signal must arrive at exactly the same moment as synaptic activity.
The purpose of the eligibility record is to connect earlier synaptic activity with a later reinforcing signal.
Fix:
Check whether the synapse remains eligible when the later signal arrives.
Check Your Understanding
A synapse participates in a neuron's firing. A reinforcing neuromodulatory signal arrives later while the synapse is still eligible. Explain what the trace records, what the later signal does, and whether the result would be described as contingent or non-contingent for an actor unit and for a critic unit.
Hints
- Start with the synapse's earlier participation.
- Explain why the later signal can act even though it arrives later.
- For the actor, include both actor input and actor output.
- For the critic, state what is absent from the trace.
Model Answer
Explain the learning sequence for the eligible synapse and classify the actor and critic traces.
Identify the record: The synapse's earlier participation leaves a synapse-specific eligibility record.
Connect the two moments: The later reinforcing neuromodulatory signal can use the record even though it did not arrive at the same time as the earlier activity.
Describe the modification: Because the synapse is eligible when reinforcement arrives, its efficacy can be modified.
Classify the actor trace: The actor trace is contingent because it depends on actor input and actor output.
Classify the critic trace: The critic trace is non-contingent because it does not involve critic output.
Earlier synaptic participation establishes eligibility, later reinforcement can modify the eligible synapse, the actor trace is contingent, and the critic trace is non-contingent.
Key Takeaways
- An eligibility trace records earlier activity at a particular synapse.
- A later reinforcing neuromodulatory signal can modify the efficacy of a synapse that is eligible.
- Actor traces are contingent because they include actor input and actor output.
- Critic traces are non-contingent because they do not include critic output.
- The mechanism connects actor-critic learning rules with Klopf's hedonistic-neuron hypothesis and corticostriatal synaptic plasticity.
Key Takeaways
- Eligibility traces preserve synapse-specific information about earlier activity.
- A later reinforcing signal can use that information to modify an eligible synapse.
- Actor traces are contingent on actor input and output, whereas critic traces do not include critic output.
- Klopf's hypothesis and corticostriatal plasticity provide biological interpretations of this timing-sensitive mechanism.