On-policy Monte Carlo Control Methods

The Exploration Problem

A control method should favor actions that currently have the highest estimated action value. However, an on-policy method does not permanently choose only that action. The policy used to make decisions remains soft, so every available action keeps a positive chance of being selected. ε-greedy action selection combines these two requirements: it strongly favors the current best-known action while retaining a probability of trying an action at random.

Splitting Probability Among Actions

An ε-greedy policy starts with the current estimated action values. The action, or actions, with the maximal estimated value are called greedy actions. Each available action receives a random-selection share of ε divided by the number of available actions. The greedy action also receives the remaining bulk of the probability. Therefore, a nongreedy action receives only its random-selection share, while the greedy action receives that same share plus the remaining preference for the best-known action.

A Four-Action Calculation

ε-soft and ε-greedy Policies

ε-soft is the broader description. It says that every available action must retain a positive chance of selection, but it does not identify one unique probability distribution. ε-greedy is a particular way to meet that requirement: give every action the same random-selection share, then place the remaining bulk on the greedy action. In this sense, ε-greedy policies are the ε-soft policies that stay closest to greedy behavior while preserving exploration.

Improving the Acting Policy

On-policy control uses the current policy to make decisions and then improves that same policy from the action-value estimates it is building. The policy is not replaced by a permanently greedy rule. Instead, it remains soft: every action stays possible, while the action with the highest current estimated value receives the strongest preference. As the estimated action values change, the identity of the greedy action, and therefore the probability distribution, can change as well.

Tracing a Policy Change

Imagine that one action is currently identified as greedy because it has the highest estimated action value. The ε-greedy policy favors that action but still gives every available action a positive chance. After returns update the estimated action values, another action may become the one with the maximal estimate. The policy then changes its preference toward that newly greedy action, while continuing to keep every action selectable.

Common Probability Mistakes

• Giving all of the remaining probability to the greedy action but forgetting its random-selection share.

  The random share is assigned to every available action, including the greedy action.

  Fix: Add the greedy action's 0.05 random share to the remaining 0.8, giving it 0.85.

• Treating ε-soft and ε-greedy as exact synonyms.

  ε-soft describes a requirement for positive selection probability, whereas ε-greedy specifies a particular distribution that satisfies that requirement.

  Fix: Describe ε-greedy as a member of the broader ε-soft policy class.

• Making the policy permanently greedy.

  An on-policy method keeps the policy soft, so every available action retains a positive chance of selection.

  Fix: Favor the greedy action while preserving the random-selection share for every action.

• Choosing the greedy action without checking the current estimates.

  The ε-greedy policy is built from the current estimated action values.

  Fix: Identify the action or actions with maximal estimated value before assigning the policy's preference.

Practice Check

Key Takeaways

1. An ε-greedy policy favors the action with the highest current estimated action value while keeping every action selectable.
2. Every available action receives a random-selection share of ε divided by the number of available actions.
3. The greedy action receives its random-selection share plus the remaining bulk of probability.
4. ε-soft is a broad policy requirement; ε-greedy is a specific probability structure within that class.
5. On-policy control improves the same soft policy that selects actions, so updated action values can change which action receives the strongest preference.