State Values

From Situations to Decisions

Suppose you know how valuable a situation is, but you do not know what each available action will do. That knowledge may not be enough to decide what to do next. State values answer the question, “How valuable is this state?” Reinforcement learning also needs the related question, “How valuable is each action from this state?”

An action value evaluates a state together with a particular action. Instead of assigning value only to a situation, an action value evaluates a state-action pair. Estimating these values gives information that can be used to suggest a policy because the values distinguish among the available actions from a state.

Choosing with a Model

State values alone can support a policy when a model is available. The model makes it possible to perform a one-step look-ahead decision: evaluate the possible next situations produced by each available action, then use their state values to select an action. In this setting, the value of a state helps guide the decision because the model supplies the missing connection between an action and the possible next state.

The model is essential in this use of state values. Without it, knowing the value of a possible successor state does not reveal which available action leads to that state. State values describe states, but they do not by themselves identify the quality of the actions that connect the current state to those states.

When the Model Is Missing

Learning from Complete Returns

Monte Carlo state-value estimation learns from complete observed returns rather than from a single outcome. Suppose an agent follows a fixed policy and repeatedly experiences episodes. To estimate the value of a particular state, record what happens after the agent visits that state and use the resulting returns as evidence about the state's value.

The state value is an expected return, while each recorded return is one observation used to estimate it. The estimation process averages the observed returns after visits to the state. As more returns are collected, their average should converge to the expected value.

For action values, the same Monte Carlo idea connects to the goal of estimating q ∗: collect returns associated with state-action pairs and use those observations to estimate how valuable the pairs are. Estimating action values is useful because it supplies action-specific information for suggesting a policy.

Counting Visits Correctly

First-visit Monte Carlo includes one return per episode containing the state. It uses the return after the first occurrence of the state and ignores later occurrences in that same episode. Every-visit Monte Carlo includes a return after every occurrence of the state. If the episode returns to the state, both the first and later returns enter the every-visit average.

Worked Return Selection

Mistakes with Value Estimates

• Treating a state value as if it directly identifies the best action.

  Without a model, state values do not identify action quality by themselves.

  Fix: Use a model for one-step look-ahead, or estimate the values of the state-action pairs directly.

• Using only one observed return as the state's value.

  The state value is an expected return, while each recorded return is only one observation used to estimate it.

  Fix: Collect observed returns and use their average as the estimate.

• Calling only the first occurrence a visit.

  Each occurrence of the state within an episode is a visit.

  Fix: Include only the first occurrence for first-visit Monte Carlo, but include every occurrence for every-visit Monte Carlo.

• Mixing first-visit and every-visit return sets.

  First-visit estimation uses one return per episode, whereas every-visit estimation uses a return after every occurrence.

  Fix: Select the returns according to the chosen visit rule before calculating the average.

Practice the Visit Rule

The first-visit selection is 4 and 6. The every-visit selection is 4, 9, and 6. The difference is not in the observed episode data; it is in the rule used to decide which visits contribute evidence to the estimate.

Key Takeaways

1. An action value evaluates a state together with a particular action.
2. A model lets state values support a one-step look-ahead decision.
3. Without a model, state values alone do not identify which action is best; action-value estimates provide action-specific information.
4. Monte Carlo state-value estimation averages complete observed returns after visits to a state.
5. First-visit Monte Carlo uses the first occurrence in each episode, while every-visit Monte Carlo uses every occurrence.