Bellman Equations

Why Values Look Ahead

The value of a state is not necessarily determined only by what happens immediately. An agent may receive a reward, move to a successor state, and then obtain additional value from that successor. The Bellman equation makes this dependency explicit: a state's value is connected to the rewards and discounted values of what may happen next.

Two Kinds of Optimal Value

Reading a Backup

A backup diagram shows the relationships used when value information is carried back toward a state. In the diagrams described by the source, open circles represent states and solid circles represent state-action pairs. Starting from a state, follow the available actions, the environment's possible responses, the resulting rewards, and the successor states. These connected outcomes provide the information used to update or express the value of the starting state.

The backup is an organizing picture, not a replacement for the equation. It helps you identify which rewards, transition probabilities, and successor-state values belong in the expected value calculation.

Building the Recursive Expression

To construct a Bellman expression, begin with the state whose value is being evaluated. List the possible outcomes after the relevant action or policy decision. For each outcome, identify its reward and the value of the successor state. Discount the future value, weight each outcome according to its probability, and combine the contributions. The result expresses the current value through the values of related future states.

current value = expected immediate reward + expected discounted successor-state value

A Small Symbolic Calculation

Using an Optimal Policy

1. Identify whether the requested quantity is v*(s) or q*(s,a).
2. Locate the state, action, and successor outcomes named by the exercise.
3. Use the stated optimal policy to determine which action or continuation is relevant.
4. Write the symbolic expression before inserting numerical values.
5. Include the immediate rewards, the probabilities of possible outcomes, the discounting of future values, and the successor-state values supplied by the problem.
6. Evaluate the expression only after the symbolic structure is clear.

This procedure is especially important for gridworld-style questions. The source describes a gridworld exercise that requires the optimal policy, equation (3.2), and a separate numerical evaluation step. The exercise asks for a symbolic expression first and then a computation to three decimal places; it states that the optimal value of the best state is 24.4 to one decimal place.

Common Interpretation Errors

• Treating v*(s) and q*(s,a) as the same quantity.

  The state-value function takes a state as its input, while the action-value function evaluates a particular action in a state.

  Fix: Check whether the exercise asks about a state alone or about a state-action pair.

• Ignoring successor states.

  The Bellman relationship connects a state's value to the discounted values of possible successor states as well as to rewards.

  Fix: Trace every possible outcome to its successor state and include the corresponding future value.

• Forgetting outcome probabilities.

  A state's value is based on an expected return, so possible outcomes contribute according to their probabilities.

  Fix: Assign each outcome its probability before combining the contributions.

• Calculating before writing the symbolic expression.

  The source distinguishes expressing the optimal value symbolically from carrying out the numerical evaluation.

  Fix: Write the equation with the optimal policy and known quantities first; evaluate it second.

• Assuming the recursive equation is only a one-step description.

  The recursive relationship defines current value through related value information for future states.

  Fix: Read the equation as a consistency relationship across the current state and possible successor states.

Practice Check

For the source's named practice settings, use the same reading strategy. The golf and recycling robot examples help frame value-function questions, while the gridworld exercise emphasizes using an optimal policy and a provided equation to express and compute an optimal value. In every case, first identify the requested viewpoint: state value, action value, or recursive value computation.

Key Takeaways

1. v*(s) is the maximum expected return from a state under an optimal policy.
2. q*(s,a) evaluates the return after taking a particular action and then following an optimal policy.
3. A Bellman equation is recursive because current value is expressed using rewards and values of possible successor states.
4. Probabilities weight possible outcomes, discounting adjusts future values, and rewards contribute directly to the expected return.
5. For symbolic exercises, identify the requested value function, apply the optimal policy, write the expression, and only then perform the numerical calculation.