Defining the Prediction Objective in Reinforcement Learning

Why State Attention Matters

A prediction objective in reinforcement learning does not treat every state as equally important by default. The relevant question is which states receive attention under the target policy π. The on-policy distribution describes how states are distributed under that policy, connecting value-prediction evaluation to the states that the target policy spends time in.

From Policy to Visited States

Start with the target policy π. The on-policy distribution is the distribution of states under that policy: it identifies which states the target policy visits and how much attention those states receive. In practical terms, this distribution is commonly described by the fraction of time spent in each state.

Reading Time Fractions

The phrase fraction of time gives an intuitive interpretation of the on-policy distribution. Imagine observing states while the target policy is being followed. A state that appears often receives a larger share of the distribution; a state that appears less often receives a smaller share. This is why the prediction objective is connected to the states on which the target policy spends time.

Continuing-Task Interpretation

In a continuing task, the process continues rather than being organized around an ending terminal state. In this setting, the on-policy distribution is interpreted as the stationary distribution under the target policy π. The key idea remains the same: the distribution describes the states associated with following π, but for a continuing task it is characterized as the stationary distribution.

Interpreting RMSVE

When evaluating an approximate value function, compare its predicted values with the true values across states. RMSVE summarizes the size of that difference in a rough overall measure. In the prediction objective, the on-policy distribution supplies the emphasis: differences in states that receive more attention under π contribute more strongly to the evaluation than differences in states receiving less attention.

Weighting the Prediction Objective

The prediction objective is weighted by the on-policy distribution. This means the objective emphasizes errors according to how states are distributed under the target policy π. The distribution is therefore not an unrelated statistic: it determines which state-wise differences between approximate and true values receive greater attention when values are evaluated.

Common Interpretation Errors

• Treating the on-policy distribution as independent of the target policy.

  The on-policy distribution describes states under the target policy π.

  Fix: Always connect the distribution to the states visited under the target policy.

• Forgetting the time-spent interpretation.

  The typical weighting is the fraction of time spent in each state.

  Fix: Ask how much time the target policy spends in each state.

• Applying only an ending-task interpretation to a continuing task.

  For continuing tasks, the on-policy distribution is the stationary distribution under π.

  Fix: Use the stationary-distribution interpretation for continuing tasks.

• Treating RMSVE as an exact explanation of why values are wrong.

  RMSVE is described as a rough measure of how much approximate values differ from true values.

  Fix: Interpret RMSVE as an overall rough difference measure, together with the state emphasis supplied by the on-policy distribution.

Check Your Understanding

Key Takeaways

1. The on-policy distribution describes states under the target policy π.
2. It is commonly understood as the fraction of time spent in each state.
3. For continuing tasks, it is the stationary distribution under π.
4. The distribution determines which states receive emphasis in the prediction objective.
5. RMSVE gives a rough measure of how much approximate values differ from true values.