Model-Based Planning
Planning and learning share value-function estimation as a central operation.
One Estimate, Two Sources
Planning and learning may appear to be separate activities, but they share an important operation: improving an estimate of how valuable situations are. The update itself does not necessarily determine whether a process is called planning or learning. The key distinction is often the source of the experience used for the update.
Learning uses experience from the real environment, while model-based planning can use experience produced by a model.
Following the Shared Record
Treat the estimated value function as a shared working record. A learning process can revise this record after receiving real experience. A planning process can revise the same record after receiving experience generated by a model. The record does not need to be duplicated simply because the experience sources differ.
Two Updates to One Estimate
A system has one estimated value function. First, it receives experience from the real environment. Later, a model produces simulated experience. How can both processes contribute to the same estimate?
Start with one estimate: The system keeps an estimated value function as a shared working record of how valuable situations are.
Use real experience: A learning process applies its update using experience from the real environment and revises the shared estimate.
Use simulated experience: A planning process applies a learning method to experience generated by a model and revises that same estimate.
Continue incrementally: Because the revisions can occur as a sequence of small updates, planning and learning can cooperate on one estimated value function.
Planning and learning can contribute to the same estimated value function. Their experience sources differ, but the update operation can remain the same.
From Model Output to Planning
A model can produce simulated experience. If a learning method is applied to that model-generated experience, the method functions as a planning method. This does not require redesigning the entire update procedure. The important change is the input source: the method receives simulated rather than real experience.
What do you think happens?
A learning method is given simulated experience produced by a model instead of experience from the real environment. What does the method become?
Reveal answer
Answer: A planning method
Applying a learning method to model-generated experience converts it into a planning method. The operation can stay the same while the source of experience changes.
Where the Boundary Blurs
Planning and learning can look like separate activities because one is associated with simulated experience and the other with real experience. In an integrated system, however, both may make incremental revisions to the same estimated value function. If the update operations are identical and only the supplied experience changes, the boundary between planning and learning becomes difficult to distinguish by the update procedure alone.
| Aspect | Learning | Model-based planning |
|---|---|---|
| Experience source | Real experience | Experience produced by a model |
| Central operation | Value-function estimation | Value-function estimation |
| Possible update style | Incremental update | Incremental update |
| Relationship to the shared estimate | Can revise the estimated value function | Can revise the same estimated value function |
When classifying a process, ask first where its experience comes from. Do not assume that planning and learning must use different update algorithms or separate value functions.
Mistakes About Planning
Assuming planning and learning must use different update procedures.
A learning method applied to simulated experience can serve as planning.
Fix:
Check the source of the experience supplied to the method.Assuming planning requires a separate estimated value function.
Incremental updates allow planning and learning to contribute to the same estimated value function.
Fix:
Understand the estimated value function as a shared working record that both processes can revise.Treating the experience source as a minor implementation detail.
The source of experience is the key distinction commonly used to separate learning from planning.
Fix:
State explicitly whether the experience comes from the real environment or from a model.
For each case, identify whether the process is learning, planning, or not distinguishable from the update operation alone: a value estimate is revised using real experience; the same method is revised using simulated experience from a model; planning and learning both make small revisions to one shared estimate.
Hints
- Focus on the source of the experience.
- Do not infer that the update algorithm must change when the source changes.
- For the shared-estimate case, separate the question of what is updated from the question of where the experience came from.
A reliable explanation of model-based planning should name both parts: the value-function update and the simulated experience supplied by a model.
Working Summary
- Planning and learning share value-function estimation as a central operation.
- Incremental updates allow both processes to contribute to one shared estimated value function.
- Learning generally uses real experience, while model-based planning uses simulated experience produced by a model.
- Applying a learning method to simulated experience turns that method into a planning method.
- The difference between planning and learning is often the source of experience, not the update operation itself.
Key Takeaways
- Value-function estimation is the common operation connecting planning and learning.
- Both processes can make incremental updates to the same estimated value function.
- Real experience is associated with learning, while model-generated simulated experience is associated with planning.
- A learning method applied to simulated experience functions as a planning method.
- To distinguish the processes, inspect the source of the experience before assuming their algorithms differ.