Actor-Critic Brain Implementation

The Central Idea

A useful first interpretation says that dopamine is the brain's reward number: one central signal that tells every learning system how much reward has occurred. The hypothetical actor-critic implementation makes a more careful proposal. The ventral striatum provides value information to the VTA and SNpc, while dopamine neurons combine value information with reward-related information. Their resulting activity corresponds to a temporal-difference error, or TD error. This is a teaching signal about how current reward-related information compares with value information, not a single master reward scalar stored in one neuron.

The Hypothetical Pathway

In the proposed arrangement, the ventral striatum sends value information to both the VTA and the SNpc. Dopamine neurons then combine value information with reward-related information. The source description is intentionally a hypothetical implementation: it identifies information flow and signal computation without reducing the whole system to one reward-producing neuron.

Tracing the Teaching Signal

A TD error is best understood here as a mismatch signal. Value information supplies one part of the comparison, while reward-related information supplies another part. When those sources differ, dopamine-related activity corresponds to the resulting TD error. The important point is not that dopamine is identical to reward; it is that dopamine-related activity reflects the relationship between value information and reward-related information.

Beyond a Master Reward

The phrase brain's reward number can create a misleading picture. That picture suggests one central scalar reward signal, located in one place, that is broadcast to every system needing reward information. The hypothetical actor-critic implementation rejects that simplification. Dopamine activity corresponds to a TD error, while the reward-related contribution is distributed across many brain areas, neurons, and input channels.

From Teaching Signal to Synapses

The actor-critic interpretation connects dopamine-related activity to learning at cortical-to-striatal synapses. In this view, the dopamine-related activity functions as a teaching-related influence on the updating or learning associated with those synaptic contacts. The proposal therefore links three levels: value information in the hypothetical arrangement, dopamine-related activity corresponding to a TD error, and changes in learning at cortical-to-striatal synapses.

Actor and Critic Roles

The critic side of the hypothetical implementation is associated with value information represented through the ventral striatum. The actor side concerns learning related to cortical-to-striatal synapses, where dopamine-related activity is hypothesized to influence updating. The TD-error-related activity connects these roles: value information contributes to the signal, and the resulting dopamine-related activity provides a teaching influence for learning.

Common Misreadings

• Treating dopamine as one scalar master reward signal.

  The hypothetical implementation says dopamine-related activity corresponds to a TD error, while the reward-related contribution is distributed across many brain areas, neurons, and input channels.

  Fix: Describe dopamine-related activity as a TD-error-related teaching signal produced from combined information.

• Treating the theoretical scalar reward and the neural implementation as identical descriptions.

  The source distinguishes the algorithmic level from the hypothetical neural level of populations, pathways, input channels, and synaptic contacts.

  Fix: State which level you are discussing: theoretical reward and TD error, or the proposed neural implementation.

• Describing value information as if it comes from nowhere in the proposed arrangement.

  The proposed arrangement states that the ventral striatum sends value information to the VTA and SNpc.

  Fix: Include the ventral striatum when explaining the value-information pathway.

• Assuming that the source specifies every detail of synaptic updating.

  The source establishes a hypothesized influence of dopamine-related activity on learning at those synapses, but does not provide a detailed update rule here.

  Fix: Use the narrower claim that dopamine-related activity influences learning or updating at cortical-to-striatal synapses.

Practice Check

Key Takeaways

1. The hypothetical implementation has the ventral striatum sending value information to the VTA and SNpc.
2. Dopamine neurons combine value information with reward-related information, producing activity corresponding to a TD error.
3. Dopamine-related activity should not be treated as one scalar master reward signal located in one neuron.
4. Reward-related contributions are distributed across many brain areas, neurons, and input channels.
5. Dopamine-related activity is hypothesized to influence learning at cortical-to-striatal synapses.