Policy Parameters and Action Selection
Contingent eligibility links presynaptic participation to postsynaptic firing.
Why Delayed Outcomes Need an Action History
An agent may choose an action and receive reward or punishment only later. In a team of learning agents, several team members may act before the outcome arrives. The learning process must therefore do more than notice the later signal: it must also retain information about which policy parameters helped produce the earlier action. A contingent eligibility trace supplies an action-linked record that makes this connection possible.
The Contingent Condition
A contingent eligibility trace links presynaptic participation to postsynaptic firing. Presynaptic input by itself is not enough. The input must participate in causing the postsynaptic neuron to fire. When that condition is met, the eligibility trace at the synapse is initiated or increased.
The word contingent identifies the condition controlling the trace. First, presynaptic activity arrives at a synapse. The relevant question is whether that activity participated in producing the postsynaptic neuron's firing. If it did not, the defining condition for a contingent increase has not been met. If it did, the trace records input that was involved in producing the neuron's response.
Checking Whether Input Is Contingent
Determine whether each event supplies the defining condition for a contingent increase in the trace.
Event A: Presynaptic input arrives, but it does not participate in causing the postsynaptic neuron to fire. The condition for a contingent increase is not met.
Event B: Presynaptic input participates in causing the postsynaptic neuron to fire. The trace at that synapse is initiated or increased.
Comparison: Both events contain presynaptic input, but only Event B contains the required participation in postsynaptic firing.
Contingency depends on the input's participation in producing postsynaptic firing, not merely on the presence of input.
Contingent and Noncontingent Records
| Feature | Contingent eligibility | Noncontingent eligibility |
|---|---|---|
| Presynaptic input | Present | Present |
| Participation in postsynaptic firing | Required | Not required by the defining condition |
| What the trace records | Input involved in producing the neuron's response | Input without the contingent participation condition |
| Connection to action history | Preserves information about involvement in producing the action | Does not provide that same contingent action-linked condition |
From Parameters to Action
A policy parameter can influence an agent's action through the activity of neuron-like units. When presynaptic activity participates in producing postsynaptic firing, the contingent trace preserves information about that action-producing involvement. In this way, the trace retains more than a record of recent input: it retains information relevant to how policy parameters contributed to the action.
Imagine a team of learning agents taking several actions before the team receives a reward. A contingent trace can preserve information about which policy parameters were involved in producing an earlier action. When the reward arrives, the learning process has an action-linked record to use when apportioning credit. If the later signal is punishment instead, the same record can help apportion blame.
Eligibility Before Credit Assignment
A contingent trace is an essential preliminary step in credit assignment, but it is not the complete answer. The trace identifies policy-parameter involvement that is relevant to an earlier action. A later reward or punishment can then be related to that action history. However, the trace does not independently establish the entire causal contribution of a parameter.
Reward Arrives After a Team Action
A team member acts, several other team members act, and the team later receives a reward. What does a contingent trace contribute, and what remains unresolved?
Action involvement: If presynaptic activity participated in producing the postsynaptic firing involved in the team member's action, the contingent trace preserves information about that involvement.
Later reward: When the reward arrives, the learning process can relate the reward to the preserved action history and use it as a basis for apportioning credit.
Remaining uncertainty: The trace alone does not determine the complete causal contribution of that team member's policy parameters. Reliable assignment requires statistical estimation across many trials.
The trace connects a later reward with an earlier action history, making credit assignment possible to investigate without claiming that the trace alone proves causation.
Common Reasoning Errors
Treating presynaptic input alone as sufficient for contingent eligibility.
The input must also participate in causing the postsynaptic neuron to fire.
Fix:
Check both parts of the condition: presynaptic input and participation in postsynaptic firing.Confusing a trace with a complete causal explanation.
The trace provides an action-linked record, but reliable credit assignment requires statistical estimation across many trials.
Fix:
Use the trace as a preliminary basis for relating outcomes to parameters, not as the final causal judgment.Limiting eligibility to rewards.
The same action-linked information can help apportion blame after punishment.
Fix:
Apply the trace to later reward or punishment.Describing the trace as unrelated to action selection.
Contingent eligibility preserves information about how policy parameters contributed to an agent's action through neuron-like activity.
Fix:
Connect the trace to the input's involvement in producing the response that helped determine the action.
Practice Check
An agent receives presynaptic input at a synapse. The input does not participate in causing the postsynaptic neuron to fire. Later, the agent receives a reward. Explain why this event does not supply the defining condition for a contingent increase. Then explain what a contingent trace would contribute if the input had participated in producing the firing.
Hints
- Focus on the condition represented by the word contingent.
- Separate the existence of input from the input's participation in producing postsynaptic firing.
- Remember that a trace preserves action history but does not by itself complete credit assignment.
What do you think happens?
Which statement best describes what a contingent eligibility trace contributes when a later punishment arrives?
Reveal answer
Answer: It preserves information about the parameter's involvement in the earlier action.
The trace gives the learning process an action-linked record that can help apportion blame. It does not independently prove the complete causal contribution, and contingent eligibility requires participation in causing postsynaptic firing.
Key Takeaways
- A contingent eligibility trace links presynaptic participation to postsynaptic firing.
- Presynaptic input alone is not enough; the input must participate in causing the postsynaptic response.
- The trace preserves information about how policy parameters contributed to an earlier action.
- Later reward or punishment can be related to that action history for credit or blame assignment.
- Eligibility is an essential preliminary step, while reliable causal assignment requires statistical estimation across many trials.
Key Takeaways
- A contingent eligibility trace links presynaptic participation to postsynaptic firing.
- The trace preserves information about policy-parameter involvement in an earlier action.
- Later reward or punishment can be related to that action-linked record.
- Eligibility supports credit or blame assignment but does not by itself establish complete causation.
- Reliable assignment requires statistical estimation across many trials.