Gridworld Value Functions

A State Is More Than a Location

In a gridworld, an agent can be described by the state it occupies, the action it chooses, the resulting transition, and the return that can be expected. Value functions organize this information by asking how valuable a situation is when decisions are made optimally.

Reading v* and q*

The key difference is what each function is evaluating. v*(s) evaluates a state by itself: it asks for the maximum expected return starting from that state when behavior is optimal. q*(s,a) evaluates a state-action pair: it asks for the maximum expected return after taking a particular action in that state and then following an optimal policy thereafter.

Following the q* Recursion

The Bellman equation for q* expresses the optimal action-value function recursively. This means that the value of a state-action pair is described using related value information rather than being treated as an isolated quantity. To work with the equation, identify the current state, identify the particular action, and then identify the optimal continuation described by the problem.

1. Locate the state named in the q* problem.
2. Locate the particular action paired with that state.
3. Read the equation supplied by the exercise rather than replacing it with an unstated alternative.
4. Identify the immediate reward and the related future-value information required by that equation.
5. Represent the optimal continuation described in the problem.
6. Evaluate only after the symbolic relationships have been made clear.

Tracing a Gridworld Calculation

A gridworld exercise connects a state, an action, a transition, and a reward. The important skill is not merely identifying a grid cell. It is tracking what happens when an action is selected and then using the provided equation and optimal policy to determine the value associated with the best state.

From Policy to Number

Practice Across Three Examples

The golf, recycling robot, and gridworld examples can be approached with the same interpretive discipline: identify the state, identify whether an action is specified, and determine whether the problem asks for a direct value interpretation or a recursive computation. What changes between examples is the description of the states, actions, rewards, and transitions. What stays consistent is the need to keep the three viewpoints separate.

Mistakes in Value-Function Reasoning

• Treating v*(s) and q*(s,a) as interchangeable.

  The two functions have different inputs and answer different questions.

  Fix: Check whether the problem concerns a state alone or a particular action in that state.

• Forgetting the optimal continuation in q*(s,a).

  q*(s,a) concerns the return after taking the particular action and following an optimal policy thereafter.

  Fix: Trace the continuation described by the problem after the specified action.

• Treating the Bellman equation for q* as a one-step calculation with no recursion.

  The equation expresses q* in terms of related value information.

  Fix: Identify both the immediate reward and the optimal continuation represented in the supplied equation.

• Substituting numbers before writing the symbolic gridworld expression.

  The gridworld exercise specifically separates symbolic expression from numerical evaluation.

  Fix: Use the optimal policy and equation (3.2) symbolically first, then calculate.

Key Takeaways

1. v*(s) is the maximum expected return from a state under an optimal policy.
2. q*(s,a) evaluates a particular action in a state and then assumes an optimal policy thereafter.
3. The Bellman equation for q* is recursive because it defines q* using related value information.
4. Gridworld symbolic calculations require the optimal policy and the provided equation before numerical evaluation.
5. In golf, recycling robot, and gridworld exercises, first determine whether the problem concerns a state, an action in a state, or a recursive value computation.