Understanding Backups in Value Prediction

Why a Backup Matters

A backup is not merely an instruction to adjust a stored number. It also specifies desired behavior for a value function. It identifies a state and indicates the numerical output that the value function should move toward for that state.

Reading the Backup Pair

The notation s ↦→ g represents a state paired with a desired numerical output. Read it as an input-output example: the input is state s, and the desired output is the number g. The estimated value for s should move toward g.

Desired Function Behavior

The backup can be viewed as a specification of what the value function should do for a particular state. Before using the backup, the function has an existing estimated output for that state. The backup supplies a desired number, and the value estimate should move toward that number.

From Backups to Function Approximation

Function approximation uses backup pairs as training examples for value prediction. Function approximation methods expect examples that describe the desired behavior of the function being approximated. In this setting, the backup supplies the example, and the resulting approximate function is interpreted as the estimated value function.

This change in viewpoint is important. A table can directly adjust one stored entry. An approximation method is not restricted to that representation, so one update can change estimated values for many states as part of the same update. It need not change only the estimate for state_B because the approximator represents value behavior across states rather than treating every state as an isolated table entry.

Learning Versus Prediction

Supervised learning and the estimated value function have different roles. Supervised learning is the machine-learning approach of learning from input-output examples. In value prediction, backups provide those examples. The estimated value function is the approximate function produced after the method processes such examples; it is then interpreted as the value function that makes the predictions.

Common Interpretation Errors

• Treating s ↦→ g as only a command to replace one stored number.

  A backup also specifies desired input-output behavior for the value function.

  Fix: Interpret s as the input state and g as the desired numerical output toward which the estimate should move.

• Assuming that function approximation must change only the estimate for the backed-up state.

  Function approximation can change estimated values for many states as part of the same update.

  Fix: Remember that the backup meaning stays the same while the approximation method can implement it through a broader function update.

• Confusing supervised learning with the estimated value function.

  Supervised learning is the approach for learning from examples, whereas the estimated value function is the resulting approximate function.

  Fix: Identify backups as training examples, supervised learning as the learning approach, and the estimated value function as the learned approximate function.

Check Your Interpretation

Key Takeaways

1. A backup specifies desired value-function behavior for a state.
2. The notation s ↦→ g pairs an input state with a desired numerical output.
3. Function approximation treats backup pairs as input-output training examples for value prediction.
4. A table can adjust one state entry while leaving other entries unchanged; an approximation update can change estimated values for many states.
5. Supervised learning is the approach that learns from examples, while the estimated value function is the resulting approximate function that produces value estimates.