Stochastic Environments
Incorrect models give planning an inaccurate description of the environment.
When Planning Trusts the Wrong Description
Planning does not examine the environment directly at every step. Instead, it relies on a model that describes how the environment behaves. If that model is inaccurate, planning can still produce a clear policy, but the policy may be poor because it was computed from the wrong description of what actions will do.
An incorrect model gives planning an inaccurate description of the environment. The resulting policy may be suboptimal because it is based on incorrectly predicted behavior.
A policy can be internally consistent and still be suboptimal. The problem may not be the planning process itself; the problem may be that planning received an inaccurate model.
Tracing a Model Error
An Action with a Missing Real-World Outcome
Suppose a model predicts that taking an action from a state leads to a favorable opportunity. Planning uses that prediction to choose a policy. In the actual environment, the favorable opportunity is not available.
Model prediction: The model represents the action as producing a favorable possibility.
Planning: Planning treats the represented advantage as an opportunity worth pursuing and includes the action in its policy.
Real interaction: The agent attempts to use the opportunity, but the environment does not provide what the model predicted.
Evidence: The failed attempt supplies evidence that the model's description needs correction.
The policy was suboptimal because it was based on a predicted opportunity that did not exist in the environment. The interaction also exposed the modeling error.
The important sequence is not simply model error followed by failure. Planning can actively bring the error to light. When a policy pursues an opportunity that exists in the model but not in the environment, the attempted use of that opportunity supplies evidence that the model needs correction.
Three Sources of Inaccuracy
Modeling errors can arise for at least three distinct reasons. The environment may behave stochastically, the model may use an imperfect function approximation, or the environment may change. These causes all produce an inaccurate model, but they describe different ways the model can fail to match reality.
| Cause | What makes the model inaccurate | What the mismatch concerns |
|---|---|---|
| Stochastic behavior | The environment can produce different outcomes, while the model may represent only one predicted behavior. | The actual outcome can differ from the model's predicted outcome. |
| Imperfect function approximation | The model's approximation does not represent the environment accurately. | The approximation fails to match the environment's behavior. |
| Environmental change | The environment changes after the model's description was formed. | The model describes behavior that is no longer the current behavior. |
Why Stochastic Behavior Misleads a Model
Stochastic behavior is a source of modeling error because the environment may produce different outcomes. A model that predicts only one outcome can therefore fail to represent what happens on a particular interaction. Planning then evaluates actions using a description that may not capture the range of behavior in the environment.
The central issue is not that planning becomes impossible. Planning can still select a policy. The issue is that the selected policy may be suboptimal when the model's predicted behavior does not match the environment's behavior.
Optimism as an Error-Discovery Mechanism
An optimistic model predicts more favorable possibilities than the environment can actually provide. It may represent greater reward or better state transitions than are really possible. Planning treats those imagined advantages as opportunities worth pursuing.
What do you think happens?
An optimistic model predicts a favorable opportunity that is not actually available. What is likely to happen when planning chooses a policy that pursues it?
Reveal answer
Answer: The opportunity is pursued, and the mismatch can provide evidence that the model is wrong.
The policy generated from the optimistic model pursues the favorable possibility. If the environment does not provide it, the attempted use exposes the mismatch and supplies evidence for correcting the model.
This is why optimistic models are especially useful for understanding error discovery. Their policies pursue favorable opportunities that may fail in reality. The failure is informative: it reveals that the model contains an opportunity the environment cannot provide, giving evidence that the model should be corrected.
Mistakes in Diagnosing Model Errors
Assuming that a clear policy must be a good policy.
A policy can be clear while still being based on incorrect predicted behavior.
Fix:
Check whether the model accurately describes the environment before judging the policy.Treating every modeling error as stochastic behavior.
Stochastic behavior, imperfect function approximation, and environmental change are distinct causes of modeling errors.
Fix:
Ask whether the mismatch comes from variable outcomes, an inaccurate representation, or a changed environment.Describing optimistic planning as guaranteed success.
Optimism predicts possibilities that may fail in reality.
Fix:
Treat a failed optimistic opportunity as evidence that can support model correction.Stopping the analysis at the policy failure.
The attempted use can reveal the model error and provide evidence for correction.
Fix:
Trace the full sequence from prediction to action, environmental outcome, evidence, and model revision.
Practice the Diagnosis
A planner chooses a policy because its model predicts a better state transition. The agent follows the policy, but the environment produces a less favorable result. Identify the model-related explanation, state whether the resulting policy can be suboptimal, and describe how the observation can help the agent.
Hints
- Start by comparing the model's predicted behavior with the environment's actual behavior.
- Use the three causes of modeling errors to identify possible sources of the mismatch.
- Explain why the observation is evidence rather than merely a failure.
Practice Solution
A planner chooses a policy because its model predicts a better state transition. The agent follows the policy, but the environment produces a less favorable result.
Identify the mismatch: The model predicts more favorable behavior than the environment provides.
Assess the policy: The policy may be suboptimal because planning selected it using the inaccurate prediction.
Classify possible causes: The mismatch may arise from stochastic behavior, imperfect function approximation, or environmental change; the observation alone does not establish which cause is responsible.
Use the observation: The less favorable result supplies evidence that the model's description needs correction.
The policy can be suboptimal because it relies on an inaccurate model, and the observed mismatch can guide correction of that model.
Key Takeaways
- Planning relies on a model rather than directly examining the environment at every step.
- An inaccurate model can cause planning to produce a suboptimal policy because the policy is based on incorrect predicted behavior.
- Modeling errors can arise from stochastic behavior, imperfect function approximation, or environmental change.
- An optimistic model represents favorable possibilities that may not exist in reality.
- When planning pursues a false opportunity, the failed attempt can reveal the model error and provide evidence for correction.
Key Takeaways
- A model is the description that planning uses to predict environmental behavior.
- If that description is inaccurate, planning can select a policy that is clear but suboptimal.
- Stochastic behavior, imperfect function approximation, and environmental change are three causes of modeling errors.
- Optimistic models can make favorable but false opportunities attractive to planning.
- Pursuing a false opportunity can expose the mismatch and supply evidence for correcting the model.