Greedy and Exploratory Policies

Two Roles for Two Policies

Off-policy learning separates the policy whose value function is being learned from the policy that generates experience. The target policy is written as π. The followed policy is written as µ. This separation allows learning to use data produced while following µ while evaluating the value function associated with π.

A Single Decision Point

Consider one state in which the target policy π selects one action, while the followed policy µ may select among several actions. The episode is generated according to µ, so the observed action comes from µ. However, the learning objective concerns π. The update therefore has to account for how the observed action relates to both policies.

The important distinction is not simply that one policy selects one action and the other selects several. The important distinction is that the observed experience was generated by µ, whereas the value being learned belongs to π. Their difference must be represented in the update.

Why the Mismatch Matters

An action in the experience has a probability under the followed policy µ and a probability under the target policy π. Off-policy learning must account for the difference between those probabilities because the data came from µ but is being used to learn about π. Ignoring that difference would treat followed-policy experience as though it directly represented the target policy.

Following an N-Step Return

An n-step return is constructed from a sequence of n actions. Therefore, the off-policy correction focuses on the same n actions that were actually used to construct that return. The correction does not need to focus on unrelated actions outside this n-step sequence.

Building ρt+n

The weighting term ρt+n combines the target-to-followed policy probability comparisons for the n actions in the return sequence. In other words, each of those n actions contributes a relative probability comparison between π and µ, and the combined weighting represents the policy mismatch for the whole n-action segment.

From TD to Off-Policy TD

The ordinary n-step TD idea uses the n-step return to correct a value estimate. In the off-policy version, the weighting term ρt+n enters that correction. The n-step return still comes from the observed sequence, but its contribution to learning is adjusted to reflect the difference between the policy that generated the sequence and the policy whose value is being learned.

Common Reasoning Errors

• Treating π and µ as the same policy.

  Off-policy learning specifically separates the target policy from the followed policy.

  Fix: Label π as the target policy and µ as the followed policy before tracing the update.

• Replacing µ with π in the observed experience.

  The experience remains data produced by µ.

  Fix: Keep µ as the source of experience and use the correction to account for the difference from π.

• Correcting for actions outside the n-step sequence.

  The n-step return uses a sequence of n actions, so the correction focuses on those same n actions.

  Fix: Trace the actions between time t and time t+n and connect the weighting to that segment.

• Using ρt+n as if it were unrelated to the TD update.

  The weighting term ρt+n is what turns the n-step TD update into an off-policy version.

  Fix: Show how ρt+n weights the value correction based on the policy mismatch.

Practice Trace

Key Takeaways

1. Off-policy learning separates the target policy π from the followed policy µ.
2. µ generates the experience, while π defines the value function being learned.
3. Because the data comes from µ but the objective concerns π, the update must account for the difference between the policies.
4. An n-step return uses n actions, so the correction focuses on those same n actions.
5. The weighting term ρt+n combines the relevant policy comparisons and turns the n-step TD update into an off-policy version.