Model-Free and Model-Based Algorithms
The model-free versus model-based distinction concerns how an action is selected.
Two Ways to Choose
The model-free versus model-based distinction concerns how an action is selected. The key question is what information appears to guide the behavior: information already stored for choosing actions, or planning ahead with a model of the environment.
Model-free algorithms are associated with cached policy or action-value information and habitual behavior. Model-based algorithms are associated with planning ahead using an environmental model and goal-directed behavior. These are two descriptions of how control over action selection can operate.
Cached Action Values
In the model-free description, information for choosing actions has already been stored. The important point is not that the agent constructs a plan at the moment of choice. Instead, cached policy or action-value information is associated with selecting the action. This is why model-free algorithms are related to habitual behavior.
A Cached Choice
Imagine an animal has repeatedly learned that pressing one particular lever is associated with a favorable outcome. Describe the choice using the model-free perspective.
Identify the stored information: The relevant information is the cached action or policy information associated with the lever action.
Identify the control style: Because the choice is described as being guided by stored action information, it illustrates the model-free side of the distinction.
Relate it to behavior: The repeated choice is an illustration of habitual behavior, which is associated with model-free algorithms.
The example represents cached decision information directly guiding a habitual action rather than planning ahead with an environmental model.
Planning with a Model
In the model-based description, the decision depends on planning ahead with a model of the environment. This approach is associated with goal-directed behavior. The distinction is therefore about the information used to select the action: cached decision information in the model-free case, or an environmental model used for planning in the model-based case.
A Planned Choice
Imagine an animal must choose between two actions and the decision is described as depending on planning ahead with a model of the environment.
Identify the decision information: The relevant information is not only an already stored action value. The description says that the agent uses a model of the environment.
Identify the control style: Planning ahead with an environmental model is associated with model-based algorithms.
Relate it to behavior: This kind of control is associated with goal-directed behavior.
The example represents model-based, goal-directed action selection.
| Question | Model-free | Model-based |
|---|---|---|
| What guides action selection? | Cached policy or action-value information | Planning ahead with an environmental model |
| Associated behavioral description | Habitual behavior | Goal-directed behavior |
| Central distinction | Stored information for choosing actions | A model used to plan ahead |
Testing Behavioral Control
Outcome-devaluation experiments help researchers investigate whether an animal's behavior is habitual or under goal-directed control. Their role is diagnostic: they provide evidence about the type of control guiding the behavior.
Reading an Outcome-Devaluation Result
An experiment compares an animal's behavior before and after the value of a usual outcome is changed. What question is this comparison designed to address?
Focus on the outcome change: The experiment changes the value of an outcome and then examines behavior in relation to that change.
Compare behavior: Researchers compare behavior before and after the outcome is devalued.
Identify the diagnostic question: The comparison provides evidence about whether the behavior is habitual or under goal-directed control.
The experiment is used to investigate the kind of control guiding the animal's behavior, not merely to label the action itself.
Common Interpretation Errors
Treating model-free as if it means having no learned information
Model-free algorithms are associated with cached policy or action-value information for choosing actions.
Fix:
Describe model-free control as action selection guided by stored decision information, associated with habitual behavior.Treating every learned action as model-based
The distinction concerns how the action is selected: cached information versus planning ahead with an environmental model.
Fix:
Ask what information appears to guide the behavior at the time of choice.Using outcome devaluation as a description of the algorithm itself
Outcome-devaluation experiments are used to provide evidence about whether behavior is habitual or under goal-directed control.
Fix:
Treat the experiment as a diagnostic method for investigating the type of control.
When analyzing a behavior, separate two questions. First, what information is available for selecting the action? Second, what kind of behavioral control does that information suggest? This keeps cached action information, environmental-model planning, habitual behavior, goal-directed behavior, and outcome-devaluation evidence conceptually distinct.
Classify each description as model-free or model-based, then state whether it is associated with habitual or goal-directed behavior: an action is selected using cached action-value information; an action is selected by planning ahead with an environmental model.
Hints
- Look for stored information already associated with choosing an action.
- Look for planning ahead with an environmental model.
Key Takeaways
- Model-free algorithms are associated with cached policy or action-value information and habitual behavior.
- Model-based algorithms are associated with planning ahead using an environmental model and goal-directed behavior.
- The central distinction is whether action selection relies on stored decision information or planning with a model of the environment.
- Outcome-devaluation experiments help researchers investigate whether an animal's behavior is habitual or under goal-directed control.
- Outcome-devaluation experiments are diagnostic: they provide evidence about the type of control guiding behavior.
Key Takeaways
- Model-free control uses cached policy or action-value information and is associated with habitual behavior.
- Model-based control uses planning ahead with an environmental model and is associated with goal-directed behavior.
- The distinction concerns how an action is selected, not simply whether the behavior was learned.
- Outcome-devaluation experiments help researchers determine what type of control appears to guide an animal's behavior.