State-Action Value Functions

Value Beyond the Current State

A value function does not need to evaluate every state as an isolated case. Under a policy π, the value of a state s is connected to what may happen after the agent is in s. The agent may receive a reward, move to a successor state, and then receive the value associated with that successor. The Bellman equation expresses this recursive consistency relationship.

Reading a Backup Relationship

A backup relationship moves value information from possible successor states back toward the state currently being evaluated. Begin with a state. The policy determines possible actions, the environment produces rewards and successor states, and the values of those successor states contribute to the value of the starting state. The word backup refers to this movement of information from later possibilities toward the current estimate.

The Recursive Bellman Relationship

The Bellman equation relates the value of a current state to the expected reward and discounted value of possible successor states. It is recursive because the value on the right-hand side includes the value function applied to states that may be reached later. Those successor-state values are evaluated using the same kind of relationship.

vπ(s) = expected value of [immediate reward + γ × value of the successor state]

For a small finite set of possible outcomes, the expectation can be written as a weighted sum. Each outcome contributes its probability multiplied by its immediate reward plus its discounted successor-state value. If the policy can select several actions, the state's value also reflects the actions selected by that policy.

States and State-Action Pairs

A state describes the situation in which the agent is being evaluated. A state-action pair adds the particular action under consideration in that state. This distinction matters in a backup diagram: an open circle identifies a state, while a solid circle identifies a state-action pair. The state connects to the actions available under the policy, and each action connects to its possible rewards and successor states.

Rewards, Probabilities, and Discounting

Three ingredients determine the contribution of a possible outcome. The immediate reward describes what is received from the transition. The successor-state value describes what is expected after that transition. The discount factor γ scales the future value, and the probability of the outcome determines how much that outcome contributes to the expectation.

A Small Numerical Backup

Consider one action from state s with two possible outcomes. Outcome 1 has probability 0.6, immediate reward 4, and successor-state value 10. Outcome 2 has probability 0.4, immediate reward 1, and successor-state value 5. Use a discount factor of 0.5. This is a generated illustration of how the expectation can combine multiple possible outcomes; it is not a numerical example taken from the source.

vπ(s) = 0.6 × (4 + 0.5 × 10) + 0.4 × (1 + 0.5 × 5) = 6.8

Mistakes in Bellman Backups

• Treating the current state as independent from future states.

  The Bellman relationship connects the current state's value to the expected reward and discounted values of possible successor states.

  Fix: For every possible outcome, include both the immediate reward and the discounted successor-state value.

• Forgetting probabilities when several outcomes are possible.

  The value is an expectation, so possible outcomes contribute according to their probabilities.

  Fix: Multiply each outcome's return by its probability before adding the contributions.

• Discounting the immediate reward instead of the future value.

  The illustrative Bellman structure adds the immediate reward to the discounted successor-state value.

  Fix: Use reward + γ × successor-state value for each outcome.

• Confusing a state with a state-action pair in a backup diagram.

  The source's diagrams distinguish open-circle states from solid-circle state-action pairs.

  Fix: Track whether each node represents the situation itself or a particular action taken in that situation.

Practice the Backup

Summary

1. The Bellman equation is a recursive consistency relationship for value functions.
2. A state's value depends on expected rewards and discounted values of possible successor states.
3. Probabilities weight the contributions of different possible outcomes.
4. Backup diagrams show how states, state-action pairs, actions, rewards, and successor states are connected.
5. The value function vπ is the unique solution to its Bellman equation.