Historical Foundations of Actor-Critic Architectures

The Delayed-Credit Problem

A central problem in reinforcement learning is connecting an earlier synaptic event with a later reinforcement signal. Miller's 1981 proposal addressed this problem with a Law-of-Effect-like learning rule containing synaptically local contingent eligibility traces. The proposal also included a critic-like sensory analyzer unit that supplied reinforcement signals to neurons.

Miller's Eligibility Trace

Miller's Law-of-Effect-like rule included a synaptically local contingent eligibility trace. In practical terms, the trace is the synapse's temporary record that an earlier activity pattern has made it eligible for a later learning change. The trace is local because it belongs to the synapse rather than being described as a single global record of every event in the system. It is contingent because the later reinforcement signal matters: the earlier event becomes relevant to learning when an appropriate consequence arrives.

The Sensory Analyzer as Critic

Miller's sensory analyzer unit was a critic-like mechanism. Its role was to provide reinforcement signals to neurons. This separates two functions that are often easy to confuse: one part of a learning system participates in or changes behavior, while another mechanism supplies information about reinforcement. In this historical proposal, the sensory analyzer occupies the second role by supplying the signal that can guide changes at learning synapses.

The Hedonistic Synapse

Seung's hedonistic synapse focuses on an individual synapse and its neurotransmitter release probability. The probability changes according to whether reward follows the synapse's release or follows its failure to release. This gives the synapse a reward-sensitive mechanism: the later reward is related to the outcome of the earlier release event, and that relationship changes how likely release is in the future.

From Historical Rules to Modern Architectures

The source connects Miller's proposal with the general features of reward-modulated spike-timing-dependent plasticity, or reward-modulated STDP, and with the later actor-critic use of temporal-difference error. The connection is functional: a local eligibility trace preserves information about an earlier synaptic event, while a later reward-related signal modulates the learning change. The actor-critic perspective then makes the division of labor more explicit: an actor is associated with behavior or action selection, while a critic supplies evaluative information. In the historical account, Miller's critic-like sensory analyzer anticipates this separation by providing reinforcement signals to neurons.

Chemotaxis as Metaphor

Bacterial chemotaxis belongs in this history as a metaphor, not as a replacement name for the learning mechanisms. The metaphor can motivate the idea that a system responds differently when conditions improve or worsen. The learning mechanisms inspired by the broader historical discussion are more specific: a synapse can retain a local eligibility trace, a reinforcement-related signal can modulate learning, and a critic-like component can provide evaluative information. Therefore, do not treat the chemotaxis metaphor itself as an eligibility trace, a hedonistic synapse, or an actor-critic method.

• Treating the bacterial chemotaxis metaphor as the learning rule itself.

  The metaphor and the synaptically local learning mechanism play different explanatory roles.

  Fix: Use chemotaxis only as an analogy, then identify the actual mechanism: a local eligibility trace combined with a later reinforcement signal.

• Treating a reward signal as if it were the eligibility trace.

  The trace marks the earlier event, while the later reinforcement signal modulates the learning change.

  Fix: Describe the trace and reinforcement as separate parts of the delayed-credit process.

• Assuming Miller's sensory analyzer and a later actor-critic critic are automatically identical.

  The source describes Miller's unit as critic-like and as providing reinforcement signals; it presents a historical anticipation, not complete identity.

  Fix: Say that the sensory analyzer anticipates the critic's evaluative role.

Check Your Understanding

1. Miller's 1981 Law-of-Effect-like rule used synaptically local contingent eligibility traces to connect earlier synaptic activity with later reinforcement. His sensory analyzer was critic-like because it supplied reinforcement signals to neurons. Seung's hedonistic synapse made neurotransmitter release probability depend on whether reward followed release or failure to release. These ideas share general features with reward-modulated STDP and anticipate the actor-critic separation between behavior-related and evaluative roles, including later use of TD error. Bacterial chemotaxis should be kept as a metaphor rather than confused with the learning mechanisms themselves.