Model-Based Reinforcement Learning in the Brain

A Choice That Changes

Imagine an animal that has learned an action leading to a valuable outcome. If that outcome becomes less valuable, a flexible animal should reconsider the action. This adjustment is characteristic of goal-directed behavior. Habitual behavior is different: an established action can continue even when its outcome is no longer as valuable. The evidence suggests that these two modes of control rely on different, though not necessarily exclusive, brain contributions.

Two Ways to Select an Action

Model-based and habitual control represent different kinds of guidance for behavior. In model-based behavior, the animal uses information about the environment and expected outcomes to support a flexible choice. In habitual behavior, a learned action can be expressed without the same degree of adjustment when outcome value changes. The contrast is functional rather than a claim that every decision follows a simple one-way sequence.

Dorsal Striatum Division

The dorsal striatum includes the dorsolateral striatum, or DLS, and the dorsomedial striatum, or DMS. The DLS is more closely associated with model-free learning and habits. The DMS is more closely associated with model-based learning and goal-directed processes.

Value, Planning, and Memory

Goal-directed choice depends on more than one brain contribution. The prefrontal cortex is implicated in executive functions such as planning and decision making. Within the prefrontal cortex, the orbitofrontal cortex, or OFC, is related both to the subjective reward value of biologically significant stimuli and to the reward expected as a consequence of actions. This makes the OFC a candidate contributor to the reward portion of an animal's environment model.

The hippocampus adds a different kind of support. It is critical for memory and spatial navigation. In rats, hippocampal function is important for navigating a maze in a goal-directed manner. This connects the hippocampus with the possibility that animals use models, or cognitive maps, when selecting actions. The hippocampus may also contribute to the human ability to imagine new experiences.

Changing the Balance

Temporary inactivation experiments provide a way to test how changing activity in a region affects behavioral control. When the DLS is inactivated, habit learning is impaired and the animal relies more on goal-directed processes. When the DMS is inactivated, goal-directed processes are impaired and the animal relies more on habit learning.

Testing the Mechanism

A common way to investigate these roles is to temporarily inactivate a brain area in a rat and then observe behavior in an outcome-devaluation experiment. The logic is comparative. If inactivating a region weakens habit learning while behavior becomes more goal directed, that region is implicated more strongly in habitual control. If inactivating another region weakens goal-directed processes while behavior becomes more habitual, that region is implicated more strongly in goal-directed control.

• Treating the DLS and DMS contrast as an absolute separation

  The evidence describes different, though not necessarily exclusive, contributions and presents the contrast as a way to organize the findings.

  Fix: Use qualified language: the DLS is more closely associated with habits and model-free learning, while the DMS is more closely associated with goal-directed and model-based processes.

• Reducing the OFC to a general movement controller

  The source emphasizes the OFC's relationship to subjective reward value and reward expected as a consequence of actions.

  Fix: Explain the OFC as a contributor of reward-value information relevant to an environment model and goal-directed choice.

• Treating the hippocampus as relevant only to memory

  The hippocampus contributes to both memory and spatial navigation, and hippocampal function is important for goal-directed maze navigation in rats.

  Fix: Connect memory and spatial navigation to the use of models or cognitive maps during action selection.

Apply the Contrast

Working Model

1. The DLS is more closely associated with model-free learning and habitual behavior.
2. The DMS is more closely associated with model-based learning and goal-directed behavior.
3. The prefrontal cortex contributes to planning and decision making, while the OFC provides information about subjective and expected reward.
4. The hippocampus supports memory and spatial navigation, linking it to cognitive maps and model-based behavior.
5. Inactivation results help reveal functional contributions: reducing DLS activity shifts behavior toward goal-directed processes, while reducing DMS activity shifts behavior toward habits.