Reinforcement Learning and Behavioral Control
Habitual behavior is primarily triggered by antecedent stimuli.
Two Ways Behavior Is Controlled
A learned behavior can be controlled in two importantly different ways. A familiar stimulus may be enough to launch a response with little further deliberation. Alternatively, behavior may depend on considering how valuable a goal is and what an action is expected to produce. These patterns are called habitual control and goal-directed control.
| Control type | What initiates selection | What matters for control | Reinforcement-learning correspondence |
|---|---|---|---|
| Habitual | An antecedent stimulus | A familiar stimulus-response pattern | Model-free |
| Goal-directed | A valued goal and expected action consequences | The relationship between an action and what it will produce | Model-based |
From Stimulus to Habit
Habitual behavior is primarily triggered by an antecedent stimulus. Once the appropriate cue is present, the learned behavior is performed more or less automatically. The key point is that control comes from the preceding stimulus, rather than from freshly evaluating the action's consequences each time.
Generated example: Imagine that a familiar signal repeatedly precedes a learned response. When the signal appears, the response may begin because the signal has become an antecedent stimulus for that behavior. The behavior is habitual when the stimulus-driven pattern, rather than a fresh evaluation of consequences, is doing the controlling.
Tracing a Goal-Directed Choice
Choosing by Goal and Consequence
Generated example: A person has a goal whose value matters and is considering an action. Explain how the action can be selected in a goal-directed way.
Identify the goal: The goal provides a value that matters to the behavior.
Consider an action: The person considers what the action is expected to produce rather than responding only to a familiar stimulus.
Relate action to consequence: The action-outcome relationship supplies information about what the action will bring about.
Select behavior: The action is guided by the value of the goal together with the expected consequence.
Goal-directed behavior is purposeful because goal value and action consequences guide the choice.
Goal-directed behavior is purposeful because it depends on two kinds of knowledge: how valuable the goal is and how an action is related to its consequences. The consequence matters to control. The behavior is therefore not characterized merely by a cue producing an automatic response; it is shaped by what the action is expected to bring about and by the value of the goal.
When Consequences Change
Environmental change exposes the difference between the two control systems. A habit can be efficient in an accustomed environment because a familiar input produces a quick response. However, reliance on that familiar pattern makes rapid adjustment difficult when the environment changes. Goal-directed control can respond more quickly because it is sensitive to changed action consequences.
What do you think happens?
A familiar stimulus still appears, but the environment no longer produces the expected consequence. Which control system is more likely to adjust its behavior rapidly?
Reveal answer
Answer: Goal-directed control
Goal-directed control is sensitive to the changed consequences of actions. Habitual control remains tied to the familiar stimulus-response pattern, which makes rapid adjustment difficult.
The Reinforcement-Learning Parallel
Reinforcement learning uses a parallel distinction. Model-free algorithms correspond to habitual behavior, while model-based algorithms correspond to goal-directed behavior. This correspondence connects an algorithmic classification in reinforcement learning with a psychological classification of learned behavior.
When relating behavioral control to reinforcement learning, keep the correspondence explicit: habitual control maps to model-free algorithms, and goal-directed control maps to model-based algorithms. Then explain the behavioral reason for the mapping: the first is tied primarily to familiar stimulus-driven performance, while the second is sensitive to goals and action consequences.
Mistakes in Applying the Distinction
Treating every learned behavior as habitual.
Learning alone does not identify the source of control.
Fix:
Ask whether an antecedent stimulus launches the response automatically or whether the action is selected using goals and consequences.Defining goal-directed behavior only as behavior with a goal.
Goal-directed control depends on both goal value and the relationship between an action and its consequences.
Fix:
Include the value of the goal and the expected consequence of the action.Assuming a habit always performs poorly.
The source describes habitual control as efficient under familiar conditions.
Fix:
Explain that the difficulty appears when the environment changes and the familiar pattern no longer fits.Reversing the model-free and model-based correspondence.
The stated reinforcement-learning parallel maps model-free to habitual and model-based to goal-directed.
Fix:
Use model-free for the habitual correspondence and model-based for the goal-directed correspondence.
Practice the Diagnosis
Generated practice: For each description, identify whether the behavior is primarily habitual or goal-directed, and then identify the corresponding reinforcement-learning classification. A familiar stimulus launches a learned response without freshly evaluating its consequences. A second behavior is selected by considering the value of a goal and what each action is expected to produce.
Hints
- Look first for the source of control: antecedent stimulus or goal and consequence.
- After identifying the behavioral pattern, apply the stated correspondence between behavioral control and reinforcement-learning classification.
Practice Answer
Classify the two generated descriptions from the practice prompt.
First description: A familiar stimulus launches the response, so the behavior is habitual. Its reinforcement-learning correspondence is model-free.
Second description: The action is selected using goal value and expected consequences, so the behavior is goal-directed. Its reinforcement-learning correspondence is model-based.
The first pattern is habitual and model-free; the second is goal-directed and model-based.
Key Takeaways
- Habitual behavior is primarily triggered by an antecedent stimulus and is performed more or less automatically.
- Goal-directed behavior is guided by the value of a goal and by the relationship between an action and its consequences.
- Environmental change reveals the difference: habitual control remains tied to familiar conditions, while goal-directed control can adjust more rapidly to changed consequences.
- Model-free reinforcement learning corresponds to habitual control.
- Model-based reinforcement learning corresponds to goal-directed control.
Key Takeaways
- Habitual control begins with an antecedent stimulus and produces a familiar response with little further deliberation.
- Goal-directed control uses goal value and action-outcome relationships to guide behavior.
- Goal-directed control adapts more quickly when environmental consequences change because it is sensitive to those consequences.
- Model-free and model-based reinforcement learning correspond, respectively, to habitual and goal-directed control.