Optimal Policies in Reinforcement Learning

The Best Possible Continuation

An optimal value function answers a focused question: starting in a particular state, what is the maximum value that can be obtained? It evaluates the best achievable outcome from that state after considering the available choices. The word optimal means that the later actions are selected to achieve the best possible result.

Reading State Values

Suppose an agent is in a particular state. Its optimal value describes the best result available from that point onward when the agent makes choices that achieve the best possible outcome. This value therefore summarizes the quality of the state under optimal decision-making. It is not merely the result of one predetermined action; it accounts for the choices available from the state.

The golf example makes this distinction concrete. An optimal value for a location answers how well the hole can be completed from that location when the available strokes are used in the best way. The value changes when the number of strokes available changes: a one-stroke contour covers only a small area near the hole, while two- and three-stroke contours reach progressively farther.

State Value versus Action Value

An optimal action-value function evaluates a state after a particular first action has been chosen. The first action is therefore part of the question. Later actions can still be selected optimally, but the initial commitment remains fixed.

In the golf example, q∗(s, driver) describes the value of state s under the commitment to use a driver first. The driver must be used for that first stroke, while later strokes may be selected optimally from the available driver or putter choices. By contrast, v∗(s) asks for the best achievable value from s without naming the first stroke in advance.

Bellman Mutual Checks

Bellman optimality equations characterize optimal values by relating the value of each state to the choices available from that state. The equations express a state-by-state consistency requirement: an assigned optimal value must agree with the best choices and the resulting values of successor states.

The recycling robot example has two states: high and low. Because there are two states, the Bellman optimality description contains two equations, one for the high state and one for the low state. Each equation checks the value assigned to its state against the choices available there. Solving the equations gives the optimal values for both states.

From Values to Policy

Once the optimal value function has been obtained, it can be used to identify an optimal policy. For each state, the policy selects actions that achieve the maximum value. The policy is therefore the action-selection side of the value calculation: the values tell us how good the best attainable outcomes are, and the policy identifies actions that attain those outcomes.

In the gridworld example, solving the Bellman equation produces v∗, the optimal value function for the gridworld states. The corresponding optimal policy identifies actions that achieve the maximum value from each state. If several actions tie for the maximum, all of them can be optimal. This is why multiple arrows in one gridworld cell represent multiple optimal actions rather than an error.

Common Interpretation Errors

• Treating the optimal value function as the value of one fixed first action.

  The optimal value function concerns the best achievable value from the state without a named first-action commitment.

  Fix: Use q∗(s, driver) when the first action is fixed as driver; use v∗(s) for the unrestricted best value from the state.

• Assuming an optimal policy must select exactly one action in every state.

  More than one action may achieve the maximum value.

  Fix: Treat all maximizing actions as optimal when they tie.

• Viewing Bellman equations as unrelated calculations for separate states.

  The equations form a system of mutual checks, with every state's value required to be consistent with the available choices.

  Fix: Consider one Bellman optimality equation for each state and solve them as a connected system.

Check Your Understanding

Key Takeaways

1. An optimal value function evaluates the best achievable outcome from a state.
2. An optimal action-value function evaluates a state after a particular first action has been chosen.
3. Bellman optimality equations relate each state's value to the available choices and characterize optimal values state by state.
4. An optimal policy selects actions that achieve the maximum value.
5. Several actions can be optimal in the same state when they tie for the maximum.