Dopamine as a Reinforcement Signal

From Actor-Critic Model to Brain Circuit

The actor-critic model separates two jobs. The actor influences which action is selected, while the critic learns value information. A proposed neural mapping assigns these jobs to different subdivisions of the striatum rather than treating the striatum as one undivided structure. This is a hypothesis and a schematic analogy, not a claim that the artificial model and the brain are identical.

The division is a division of emphasis, not a claim that one subdivision receives only one kind of information. The dorsal striatum is primarily associated with influencing action selection. The ventral striatum is associated with value learning and reward processing, including assigning affective value to sensations. In this proposed mapping, the actor is associated with the dorsal striatum, while the value-learning part of the critic is associated with the ventral striatum.

Value and Reward Become Dopamine Activity

To reconstruct the proposed mechanism, begin with information about a situation. The cerebral cortex, together with other structures, sends the striatum information about stimuli, internal states, and motor activity. The ventral striatum supplies value information to dopamine neurons in the ventral tegmental area and substantia nigra pars compacta. These dopamine neurons combine value information with information about reward. Their resulting activity corresponds to the temporal-difference error signal in the model.

Dopamine at Cortical-Striatal Synapses

Dopamine is not presented as an isolated output. Dopamine neurons have widely branching axons, and those axons make synaptic contact with spines on the dendrites of medium spiny neurons. Medium spiny neurons are the main input/output neurons of both the dorsal and ventral striatum. Cortical axons also contact the tips of these spines.

The proposed learning rule is located at the relationship between cortical input and striatal neurons. Cortical axons provide input at dendritic spines, while dopamine axons also contact those spines. According to the hypothesis, the effectiveness of cortical-to-striatal synapses can change in a way that depends critically on dopamine reinforcement. The timing and presence of dopamine therefore provide a way for an outcome-related signal to influence learning at connections involved in striatal processing.

From Dopamine Activity to Repeated Action

A reinforcement signal helps explain why an animal repeats one behavior instead of another. In the proposed account, the dorsal striatum is associated with influencing action selection, while dopamine activity supplies reinforcement that can affect learning at striatal synapses. If dopamine activity follows an instrumental behavior, the learning process can make the relationship between the relevant cortical input and striatal response more effective. The behavior can then become more likely to be selected again.

Evidence from Self-Stimulation

In the Olds and Milner experiment from 1954, rats could press a lever to receive electrical stimulation. The rats learned to press the lever, demonstrating that electrical stimulation of relevant brain systems could reinforce an instrumental behavior. This experiment supplied an early behavioral demonstration that an animal may repeat an action when that action produces a reinforcing brain event.

The important inference is behavioral: the rats learned an instrumental response because the response produced electrical stimulation. The experiment did not by itself establish every detail of the dopamine actor-critic mapping. It showed that stimulation of brain systems can function as a reinforcing consequence for lever pressing.

Optogenetic Evidence and Species Limits

Optogenetic experiments provide causal evidence by using light-sensitive proteins and laser flashes to control selected dopamine neurons. These experiments show that phasic dopamine activation can produce learned preferences and influence whether responding and learning continue. This supports the interpretation that phasic dopamine activity can act like a reinforcement signal in control and prediction.

• Treating the actor-critic mapping as a proven one-to-one identity between an artificial model and the brain.

  The mapping is explicitly proposed and schematic. It assigns different emphases to striatal subdivisions rather than claiming that the brain and model are identical.

  Fix: Describe the dorsal striatum as associated with action selection and the ventral striatum as associated with value learning and reward processing in the proposed mapping.

• Saying that dopamine activity is proven to be exactly the temporal-difference error.

  The source says dopamine activity corresponds to the model's signal and acts like a reinforcement signal, but does not claim exact identity.

  Fix: Use the more careful wording that dopamine activity is a proposed neural counterpart or correspondence to the temporal-difference error signal.

• Claiming that the Olds and Milner experiment directly proved the complete dopamine mechanism.

  The experiment demonstrated reinforcement by electrical stimulation and does not, by itself, establish every part of the proposed circuit.

  Fix: Use it as behavioral evidence that brain stimulation can reinforce an instrumental response.

• Inventing different mouse, rat, and fruit-fly outcomes when the supplied material does not list them.

  The source pack mentions the species but does not provide enough detail for a species-by-species comparison.

  Fix: State only that the evidence base includes those species and that the supplied material does not specify the differences.

Check Your Understanding