Value Prediction in Reinforcement Learning

Forecasting Future Reward

A reinforcement learning agent often has to make decisions before it knows the complete result of those decisions. Value prediction gives the agent a forecast: how much total reward might be available in the future? A value estimate is therefore a long-run prediction of total future reward, not merely a description of the next immediate result.

The challenge becomes larger when an agent must predict values for many states and actions. Treating every prediction as an entirely separate case may not make good use of experience. Function approximation connects reinforcement learning with generalization: information from some states or actions can support value estimates for other states or actions. The central process is to use a reinforcement learning method to produce value-prediction backups, use those backups as training examples, and then evaluate the resulting approximation.

From Backup to Training Example

A useful way to follow the information is as a three-part chain. First, a reinforcement learning value-prediction method produces backups. Next, those backups serve as training examples for a function approximation method. Finally, the approximation is judged with a performance measure such as MSVE. The direction matters: MSVE evaluates the resulting approximation; it does not produce the training examples.

Generalization Across States

Function approximation is useful because a reinforcement learning system may need values for many states and actions. An approximation method can connect these predictions instead of treating every one as an entirely separate case. Experience associated with some states or actions can therefore contribute to estimating values for other states or actions.

Evaluating the Approximation

Producing predictions is not enough to establish that an approximation method is performing well. The method is also considered through a measure such as MSVE. In practical reasoning, ask two separate questions: what information trains the approximation, and how is the resulting approximation evaluated? Backups answer the first question by providing training examples. MSVE helps address the second by measuring performance.

Why Gradient Methods Matter

There is no single universal function approximation method for value prediction. The possible range is large, and the source material notes that too little is known about many alternatives to support a reliable evaluation or recommendation of all of them. For that reason, the discussion focuses on methods based on gradient principles.

Gradient-descent methods are one class of function approximation methods. Linear gradient-descent methods receive particular attention because they are simple, considered promising, and useful for exposing important theoretical issues. Their importance is therefore both practical and explanatory: their relative simplicity makes underlying questions easier to study.

State Values and Action Values

The distinction is about what is being evaluated. V asks how promising a state is for the agent. Q asks how promising a particular action is. Both are long-run predictions rather than descriptions limited to the next immediate result.

Mistakes in Value Reasoning

• Treating a value estimate as an immediate reward

  A value estimate is a long-run prediction of total future reward.

  Fix: Interpret 8 as the predicted total future reward associated with that state or action, depending on whether the estimate is V or Q.

• Confusing V with Q

  V assigns an estimate to a state, while Q assigns an estimate to a specific action.

  Fix: Ask what is being evaluated: the situation itself or the particular action taken in that situation.

• Reversing training and evaluation

  Backups provide training examples; MSVE evaluates the performance of the approximation method.

  Fix: Keep the sequence as backup, training example, approximation, then MSVE evaluation.

• Assuming gradient-based methods are the only possible approximation methods

  Gradient-based methods are a focused subset of a much larger design space.

  Fix: Explain that gradient-based methods receive attention because they are promising and theoretically revealing.

Practice the Decision

Key Takeaways

1. Function approximation helps reinforcement learning generalize value predictions across many states and actions.
2. Reinforcement learning backups can become training examples for a function approximation method.
3. MSVE evaluates the performance of the resulting approximation; it belongs to the evaluation stage.
4. Gradient-based methods are a focused subset of a much larger design space, with linear gradient-descent methods receiving attention because they are simple, promising, and theoretically useful.
5. V estimates the long-run reward from a state, while Q estimates the long-run reward from taking a particular action.
6. Using V means comparing the values of states reached by actions; using Q means comparing the action-value estimates directly.