Reinforcement Learning Problems

The Agent's Starting Knowledge

Two reinforcement learning problems may involve similar agents and environments but still require different approaches. The decisive question is not only what the environment is like. It is what the agent already knows about how that environment behaves.

Reading an Environment Dynamics Model

A model of an environment's dynamics describes what can happen after the agent takes an allowable action in a state. In an MDP, the model contains two kinds of information for all states and allowable actions: one-step transition probabilities and expected rewards.

The model therefore connects a current state and an allowable action with information about the next state and the reward expected from that one-step interaction. To count as complete knowledge, this information must cover the relevant states and allowable actions and must be complete and accurate.

Complete Knowledge

For an MDP, completeness means that the model covers all states and allowable actions relevant to the problem. It supplies both one-step transition probabilities and expected rewards. Accuracy matters as well: a model that is available but not correct does not satisfy the requirement for complete knowledge.

Incomplete Knowledge

The model may be unavailable as a complete source of information about the relevant states, allowable actions, transition probabilities, and expected rewards. The key point is the absence of a complete and perfect model, not simply the fact that the agent knows little in a general sense.

Classifying by Initial Information

To classify a problem, inspect what the agent knows at the start. Do not classify it only from the environment's description. The same kind of environment can belong to different categories if one agent begins with a complete and accurate model while another does not.

• Treating any model as complete knowledge.

  Complete knowledge requires a complete and accurate model covering the relevant states and allowable actions and supplying transition-probability and expected-reward information.

  Fix: Check completeness and accuracy rather than merely checking whether some model information exists.

• Classifying the problem from the environment alone.

  The classification depends on what the agent initially knows about the environment's behavior.

  Fix: Inspect the model available to the agent at the beginning of the problem.

• Ignoring expected rewards.

  In an MDP, the model contains both one-step transition probabilities and expected rewards for all states and allowable actions.

  Fix: Verify that both kinds of information are included.

• Ignoring model accuracy.

  Complete knowledge requires a complete and accurate dynamics model.

  Fix: Treat accuracy as a required part of the classification.

Classification Checklist

1. Identify the model of the environment's dynamics available to the agent at the start.
2. Check whether the model covers the relevant states and allowable actions.
3. Check whether it provides one-step transition probabilities and expected rewards.
4. Check whether the model is complete and accurate.
5. Classify the problem as complete knowledge only when the complete and accurate model is available; otherwise classify it as incomplete knowledge.

1. Complete knowledge means that the agent begins with a complete and accurate dynamics model. In an MDP, that model includes one-step transition probabilities and expected rewards for all states and allowable actions. Incomplete knowledge means that a complete and perfect model is unavailable. The classification depends on the agent's initial information, not merely on the environment itself.