Model-Based Policy Computation

When Planning Trusts the Wrong Description

Planning does not examine the environment directly at every step. Instead, it relies on a model that describes how the environment behaves. If that description is inaccurate, planning can still produce a clear policy, but the policy may be poor because it was computed from incorrect predictions.

A Model Prediction Meets Reality

The important issue is not merely that a model error exists. The policy exposes the error when it acts on an opportunity that the model represents but the environment cannot actually provide. The observed mismatch is evidence that the model needs correction.

Why Models Become Inaccurate

Modeling errors can arise in several ways. The environment may behave stochastically, so its behavior cannot be represented perfectly by a simple prediction. A function used to approximate the environment may be imperfect, so the model's representation is inaccurate. The environment may also change, making a model that was previously useful no longer match current behavior.

Outdated Models Over Time

Environmental change creates a time-dependent modeling problem. A model can once have described the environment accurately, yet later become inaccurate because the environment no longer behaves as it did when the model was formed. Planning then uses an outdated description and may compute a policy that is no longer suitable.

Optimism as an Error Detector

An optimistic model predicts more favorable possibilities than the environment can actually provide. It may represent greater reward or better state transitions than are really possible. Planning treats these imagined advantages as opportunities worth pursuing, so the resulting policy is likely to try them.

This creates a useful learning sequence: the optimistic model proposes an attractive opportunity, planning produces a policy that pursues it, the environment fails to deliver the predicted result, and that attempt provides evidence for correcting the model. Planning is therefore not only a way to use a model; when its policy tests optimistic predictions, it can also help expose model errors.

Check the Causal Chain

• Assuming that a clear policy must be a good policy.

  A policy can be clear while still being based on incorrect predicted behavior.

  Fix: Evaluate the accuracy of the model that planning used, not only the clarity of the resulting policy.

• Treating model error as inevitable failure with no learning value.

  The attempted opportunity supplies evidence that the model needs correction.

  Fix: Use the mismatch between prediction and observation to identify and correct the model's error.

• Looking for only one source of modeling error.

  Modeling errors can also arise from stochastic behavior and environmental change.

  Fix: Check for stochastic behavior, imperfect function approximation, and changes in the environment.

• Assuming that optimism means the favorable outcome is guaranteed.

  Optimism means the model predicts more favorable possibilities; reality can disconfirm them.

  Fix: Treat optimistic opportunities as predictions to be tested, not as established facts.

Key Takeaways

1. Planning relies on a model rather than examining the environment directly at every step.
2. An incorrect model gives planning an inaccurate description of the environment, so the resulting policy may be suboptimal.
3. Modeling errors can arise from stochastic behavior, imperfect function approximation, or environmental change.
4. An optimistic model represents favorable possibilities that may not exist in reality.
5. When a policy pursues an optimistic opportunity and the environment disproves it, the observation can reveal the model's error and support correction.