Planning with Models

Simulated Experience

In reinforcement learning, an agent does not always need to obtain every learning experience directly from the environment. A model can imitate the environment well enough to produce possible experience for inspection or planning. This changes the source of the experience: instead of relying only on a direct interaction, the system can ask the model what might happen.

One Simulated Transition

A transition is one simulated step from a starting state after an action is supplied. To interpret this request, focus on two pieces of input: the state and the action. The model uses that state-action information to describe what can happen next.

A sample model answers with a single possible way that the step could unfold. A distribution model answers more broadly by generating all possible transitions and attaching their probabilities of occurring. Both model types describe a transition, but they provide different amounts of information about its possible outcomes.

Sample and Distribution Outputs

Generating an Episode

A simulation can continue beyond one transition. When the input consists of a starting state and a policy, the model can extend the simulation into an episode. The policy supplies the actions used while the model supplies possible experience resulting from those actions.

A sample model can produce an entire episode as one possible sequence of experience. In contrast, a distribution model can generate all possible episodes and the probabilities of those episodes. Therefore, a sample-model episode is narrow and path-specific, while a distribution-model result describes the broader set of possible episodes in probabilistic terms.

Choosing the Right Interpretation

When interpreting a model's output, use two questions in order. First, identify the requested input. A state and action describe a step, so interpret the result as a transition. A state and policy describe an extended simulation, so interpret the result as an episode. Second, identify the model type. A sample model gives one possible result; a distribution model gives every possible result together with its probabilities.

1. Read the input: state and action means transition; state and policy means episode.
2. Read the model type: sample means one possible result; distribution means all possible results with probabilities.
3. State the output at the same scale as the input: one transition or one episode for a sample model, and all transitions or all episodes with probabilities for a distribution model.

Common Interpretation Errors

• Treating a sample model as if it lists every possible outcome.

  A sample model returns one possible transition or episode, not all possible results.

  Fix: Interpret the output as one possible result unless the model is identified as a distribution model.

• Treating a distribution model as if it returns only one outcome.

  A distribution model returns all possible transitions or episodes together with their probabilities.

  Fix: Look for the complete probability-weighted collection of possible results.

• Using the wrong simulation scale.

  A state and policy describe an episode-level simulation, while a state and action describe a transition-level simulation.

  Fix: Identify the requested input before interpreting the output.

• Ignoring the policy when describing an episode.

  The episode-level input consists of a starting state and a policy.

  Fix: Include both the starting state and the policy when explaining how the episode is generated.

Key Takeaways

1. A model lets a reinforcement-learning system simulate possible experience instead of relying only on directly observed experience.
2. A state and action describe one simulated transition.
3. A state and policy describe an episode-level simulation.
4. A sample model returns one possible transition or episode.
5. A distribution model returns all possible transitions or episodes together with their probabilities.