Actor and Critic Learning Rules Explained
An eligibility trace is a temporary synaptic record that lets later reinforcement assign credit to earlier activity.
The Delayed-Credit Problem
A common first guess is that a synapse should change as soon as an action is produced. The difficulty is that the reinforcement signal evaluating that action may arrive later. At that later point, the learning system still needs a record of which synapses helped produce the action. An eligibility trace provides that temporary synaptic record.
A Trace That Waits
An eligibility trace is a temporary synaptic record. Earlier activity creates or increments the record, and the record remains available long enough for a later reinforcement signal to use it. If the trace is still non-zero when reinforcement arrives, the synapse is eligible to receive credit for the reinforced action.
Credit for an Earlier Action
An actor synapse participates in producing an action. The reinforcement signal evaluating that action arrives later, while the synapse's eligibility trace is still non-zero. How is credit assigned?
Earlier activity: Activity in the actor helps produce the action, and the relevant interaction at the synapse creates or increments its eligibility trace.
Temporary record: The trace preserves information about the synapse's earlier participation instead of requiring the reinforcement signal to arrive at the exact same moment.
Later reinforcement: The reinforcement signal arrives while the trace is still non-zero.
Credit assignment: Because the trace is available, the synapse is eligible for modification and can receive credit for the reinforced action.
The later reinforcement signal can modify the earlier participating synapse through its eligibility trace.
Actor Traces and Hebbian Change
The actor's eligibility trace is contingent: it depends on both presynaptic and postsynaptic factors. This connects the trace to the activity relationship emphasized by Hebb's proposal about synaptic change. In the actor rule, a synapse becomes eligible through the relevant interaction of activity before reinforcement is applied. The reinforcement signal then supplies the additional learning factor that determines whether that eligible synapse receives credit.
When explaining an actor trace, always name both activity factors. Calling it merely a record of presynaptic activity loses the defining feature of the actor learning rule: the trace is contingent on presynaptic and postsynaptic factors together.
Critic and Actor Compared
| Learning rule | Eligibility information | Learning-factor description |
|---|---|---|
| Critic | Depends only on presynaptic signals | Two-factor learning rule: reinforcement interacts with the presynaptic trace |
| Actor | Depends on presynaptic and postsynaptic activity | Three-factor learning rule: reinforcement combines with a trace reflecting two kinds of neural activity |
Timing and Causal Order
The order of activity matters because the presynaptic factor must help cause the postsynaptic factor for credit assignment to be meaningful. Neural activation is also not instantaneous in realistic neural models. Therefore, forming a contingent actor trace requires attention to when the relevant activity becomes active, not merely whether both factors appear somewhere in the episode.
Common Reasoning Errors
Assuming the synapse must change immediately when the action is produced.
This ignores the delayed-credit problem in actor-critic learning.
Fix:
Use the eligibility trace as a temporary record that remains available for the later reinforcement signal.Describing the actor trace as presynaptic-only.
The actor trace depends on both presynaptic and postsynaptic factors.
Fix:
Identify the actor trace as contingent on both kinds of activity.Treating the critic and actor traces as identical.
The critic trace depends only on presynaptic signals, whereas the actor trace depends on presynaptic and postsynaptic activity.
Fix:
Use the presynaptic-only versus contingent distinction to compare the rules.Ignoring activation time and causal order.
The presynaptic factor must help cause the postsynaptic factor, and realistic neural activation is not instantaneous.
Fix:
Consider the order and activation time of the factors when interpreting a contingent trace.
Check Your Understanding
A synapse in the actor helps produce an action. Presynaptic activity contributes to postsynaptic activity, and the actor's contingent trace is then available. Reinforcement arrives later. Explain why this synapse can receive credit, and state which two activity factors made its actor trace different from a critic trace.
Hints
- Start with the definition of an eligibility trace.
- Identify what happens before reinforcement arrives.
- Compare the information used by the actor and critic traces.
What do you think happens?
Which statement best describes the actor's eligibility trace?
Reveal answer
Answer: It depends on presynaptic and postsynaptic activity, with reinforcement arriving later.
The actor trace is contingent on both presynaptic and postsynaptic factors. It preserves eligibility so later reinforcement can assign credit to earlier activity.
Key Takeaways
- An eligibility trace is a temporary synaptic record that connects earlier activity to later reinforcement.
- The actor trace is contingent on both presynaptic and postsynaptic factors.
- The critic trace depends only on presynaptic signals, while the actor trace reflects two kinds of neural activity.
- For actor credit assignment to be meaningful, the presynaptic factor must help cause the postsynaptic factor.
- Because neural activation is not instantaneous in realistic models, the timing and causal order of activity must be considered.
Key Takeaways
- Eligibility traces preserve a temporary record of earlier synaptic activity until a later reinforcement signal arrives.
- The actor uses a contingent trace formed from presynaptic and postsynaptic factors.
- The critic uses a presynaptic-only trace, making its learning interaction different from the actor's.
- Causal order and activation time determine whether activity provides meaningful credit for a later reinforced action.