Reinforcement Learning Methods

From Parameters to Behavior

Many reinforcement learning discussions begin by asking how good a situation or action is. A policy gradient method begins from a different angle: it searches directly through policies. Each policy is described by a collection of numerical parameters. The method uses experience from interaction with the environment to estimate how those parameters should change so that the policy performs better.

The Interaction Loop

A policy gradient method cannot decide how its parameters should move from the parameters alone. It needs information produced by the agent's interaction with the environment. The current parameters define the policy being used. The agent follows that policy, interacts with the environment, and obtains behavioral information from the resulting interaction. That information is used to estimate a direction for improving the policy. The parameters can then be adjusted, producing another parameter-defined policy.

Improving the Gradient Estimate

The central estimate in a policy gradient method is a direction for changing the policy's numerical parameters. Some methods improve this estimate with a value-function estimate. The value estimate can make the estimated improvement direction more useful, but it is not the same thing as the policy gradient estimate.

Backup Dimensions

Reinforcement learning methods can be organized along several independent dimensions. First ask how information is obtained: a sample backup uses a sampled trajectory, whereas a full backup uses a distribution of possible trajectories and requires a model. Then ask how far the backup looks ahead. Finally ask how the value function is represented. These questions should remain separate because one choice does not determine the others.

Backup kind describes breadth, not distance. A sample backup may use one observed path, while a full backup may account for possible paths. Backup depth is a separate question. It ranges from one-step temporal-difference updates, through intermediate n-step and mixed methods, to full-return Monte Carlo updates.

Backup depth also describes the amount of bootstrapping. One-step temporal-difference backups rely strongly on an estimated value after the immediate step. Full-return Monte Carlo backups look through the complete return instead. Intermediate methods occupy the space between these choices.

Representation Spectrum

A third dimension concerns how the value function is represented. The spectrum runs from tabular representation through state aggregation and linear methods to nonlinear methods. This representation choice is independent of backup kind and backup depth. For example, two methods can use the same backup idea while representing the value function in different ways.

Search Extremes

The depth dimension also helps position search methods. At one extreme, one-step temporal-difference backups use an estimated value after the immediate step. At the other extreme, exhaustive search carries full backups to terminal states. In a continuing task, search can continue until discounting makes later rewards contribute negligibly. Search methods between these extremes may look ahead only to a limited depth and may select which possibilities to explore.

Classifying Hypothetical Methods

Common Classification Mistakes

• Treating a value function as required by every policy gradient method.

  The source states that some policy gradient methods do not appeal to value functions.

  Fix: Say that value-function estimates can improve gradient estimates in some methods, but are not required in all methods.

• Confusing the policy with the value estimate.

  The primary search object is a policy defined by numerical parameters.

  Fix: Track the policy parameters and the direction estimated for changing them.

• Treating sample backups and full backups as different backup depths.

  Backup kind concerns breadth of information, while backup depth concerns how far ahead the backup looks.

  Fix: Classify information source and lookahead depth separately.

• Assuming one-step and full-return backups are the only possible depths.

  The source describes intermediate methods between one-step temporal-difference and full-return Monte Carlo updates.

  Fix: Place intermediate methods between the two endpoints.

• Using one label to describe every dimension of a method.

  Methods can differ independently in information source, lookahead depth, and value-function representation.

  Fix: Use the three-pass procedure: information source, depth, representation.

Practice: Three-Pass Classification

Method Comparison Checklist

1. Policy gradient methods search through policies defined by numerical parameters.
2. Environment interaction provides the information used to estimate a direction for changing those parameters.
3. Value-function estimates can improve gradient estimates in some methods, but they are not required in every policy gradient method.
4. Sample backups use sampled trajectories; full backups use a distribution of possible trajectories and require a model.
5. Backup depth ranges from one-step temporal-difference updates through intermediate methods to full-return Monte Carlo updates, with bootstrapping decreasing as the backup reaches further into the return.
6. Value-function representation ranges from tabular methods through state aggregation and linear methods to nonlinear methods, so methods should be compared across independent dimensions.