Actor-Critic Reinforcement Learning

From Algorithm to Synapse

Actor-critic reinforcement learning is often introduced as an abstract system with two roles: an actor that chooses actions and a critic that evaluates learning-relevant information. A neural interpretation asks a more specific question: what changes at a synapse when these roles are implemented by biological activity? In the hypothetical implementation described here, actor and critic learning rules correspond to different forms of plasticity at corticostriatal synapses.

The actor and critic are not simply two names for the same learner. The actor is associated with selecting actions. The critic is associated with evaluating information that is relevant to learning. In the neural account, the important distinction is preserved by the activity records retained by their synapses: the actor and critic use different eligibility traces.

Eligibility Before the Update

An eligibility trace is a synapse-specific record of past activity. It allows a synapse to retain information about an earlier event until a neuromodulatory signal arrives. This matters because the signal that influences learning does not necessarily arrive at the same moment as the original presynaptic and postsynaptic activity.

Timing Shapes STDP

Spike-timing-dependent plasticity, or STDP, makes synaptic change depend on the relative timing of presynaptic and postsynaptic activity. The synapse therefore responds not merely to whether activity occurred, but to the relationship between activity arriving before the synapse and activity in the receiving neuron. Both the order of the spikes and the time difference between them are part of the learning rule.

Dopamine Converts Eligibility

Reward-modulated STDP adds a neuromodulatory condition to timing-based plasticity. First, the synapse experiences the activity relationship required for STDP. A neuromodulator such as dopamine must then arrive within an appropriate time window for the reward-modulated rule to apply. The source describes this window as lasting up to 10 seconds after the conditions for STDP are met.

Corticostriatal Mapping

Corticostriatal synapses convey signals from the cortex to principal neurons in dorsal and ventral subdivisions of the striatum. In the hypothetical actor-critic implementation, these synapses receive neural activity and dopamine-related neuromodulatory input. The actor-critic distinction is therefore expressed through the plasticity rule and through the kind of activity record retained by the synapse.

This mapping should be read as a conceptual correspondence, not as a claim that every brain implements exactly this architecture. Cortex-to-striatum signals provide the corticostriatal activity context. The actor and critic represent different learning roles. Dopamine-related input supplies neuromodulation. Eligibility traces preserve the relevant past activity so that the plasticity rule can connect earlier activity with later neuromodulation.

The Learning Cycle

Consider a conceptual trial in which cortical activity reaches corticostriatal synapses. The actor-associated activity contributes to choosing an action, while the critic-associated activity supports evaluation. The synapses retain their appropriate eligibility traces. If the relevant dopamine-related signal arrives within the appropriate window, the earlier activity can participate in reward-modulated plasticity. The actor and critic are thus connected to synaptic learning through different records of activity rather than through a single undifferentiated update.

Mistakes in the Mapping

• Treating the actor and critic as identical learners

  The source distinguishes them by their roles and by the eligibility traces they use.

  Fix: Associate the actor with action choice and a contingent trace; associate the critic with evaluation and a non-contingent trace.

• Ignoring spike timing in STDP

  STDP makes the relative timing and time difference between the activities part of the learning rule.

  Fix: Ask about the order and temporal relationship of the presynaptic and postsynaptic activity.

• Assuming dopamine must arrive simultaneously with the spikes

  The source describes dopamine arriving within an appropriate window, lasting up to 10 seconds after the STDP conditions are met.

  Fix: Separate the time of the original activity from the later neuromodulatory window.

• Confusing an eligibility trace with the final synaptic update

  The trace preserves past activity so that a later neuromodulatory signal can interact with it.

  Fix: Describe the trace as retained eligibility and dopamine-related input as the later modulatory condition.

• Presenting the neural implementation as established for every brain

  The source explicitly presents the walkthrough as a hypothetical neural implementation.

  Fix: Describe it as a conceptual mapping whose plausibility is supported by evidence for reward-modulated STDP at corticostriatal synapses.

Key Takeaways

1. The actor is associated with choosing actions, while the critic is associated with evaluating learning-relevant information.
2. STDP makes synaptic change depend on the order and time difference between presynaptic and postsynaptic activity.
3. Reward-modulated STDP combines an STDP-related activity condition with dopamine arriving within an appropriate time window, described here as up to 10 seconds.
4. Eligibility traces preserve synapse-specific records of past activity; the actor uses a contingent trace, while the critic uses a non-contingent trace.
5. Corticostriatal plasticity provides a hypothetical neural implementation in which actor-critic learning rules correspond to synaptic change.