Value Functions and State Values

A Backup Means More Than an Update

A backup is often described as an instruction for changing an estimated value. There is another important way to read it: a backup also specifies how the value function should behave. When a backup is written as s ↦→ g, it pairs a state s with a desired numerical output g. The value function should produce an estimate for state s that moves toward g.

Reading the Backup Pair

Suppose the backup is state_A ↦→ 8. The left side identifies the state whose value is being addressed. The right side supplies the numerical target toward which the estimated output should move. Together, the two sides describe desired input-output behavior for the value function, not merely a number to be stored somewhere.

The pair can therefore be used as a training example. It tells a learning method what output is wanted when the input is state_A. The pair does not by itself say that every implementation must replace the current estimate with 8 immediately. It specifies the desired behavior toward which the estimate should move.

Function Approximation for Prediction

Function approximation uses backup pairs as training examples for value prediction. A function approximator receives the state as an input and produces an estimated numerical value as its output. The backup pair provides evidence about the output that the approximator should learn to reproduce for that state.

Table Adjustment and Approximation

With a table, each state has its own stored entry, so a direct implementation can adjust the entry for s while leaving other entries unchanged. Function approximation broadens the implementation. The method can change estimated values for many states as part of the same update. The backup still concerns the particular pair s ↦→ g, but the approximator may use that evidence to alter more than one prediction.

Learning Process and Learned Function

Supervised learning is the machine-learning approach of learning from input-output examples. In this setting, backup pairs supply those examples. The estimated value function is not the learning approach itself. It is the resulting approximate function, interpreted as the function that predicts values for states. Keeping these roles separate prevents a common confusion: supervised learning describes how the function is learned, while the estimated value function describes what makes the predictions.

Mistakes in Interpreting Backups

• Treating s ↦→ g as only a command to replace one stored number.

  The backup also specifies desired input-output behavior. It says that the estimated output for state_A should move toward 8.

  Fix: First identify the state input and desired output. Then consider whether the implementation is a table adjustment or a function-approximation update.

• Assuming function approximation must change only the estimate for the backed-up state.

  A function-approximation method can change estimated values for many states as part of the same update.

  Fix: Preserve the meaning of the backup while allowing the approximation method to implement it in a broader way.

• Confusing supervised learning with the estimated value function.

  Supervised learning is the approach of learning from input-output examples, whereas the estimated value function is the approximate function interpreted as making value predictions.

  Fix: Describe backup pairs as training examples, supervised learning as the learning approach, and the estimated value function as the resulting predictor.

Check Your Interpretation

The Central Distinction

1. A backup of the form s ↦→ g pairs a state input with a desired numerical output.
2. The pair can be treated as an input-output training example for value prediction.
3. With a value table, a backup can shift one state entry partway toward its target while leaving other entries unchanged.
4. With function approximation, the same backup meaning can be implemented by an update that changes estimated values for many states.
5. Supervised learning is the approach that learns from backup pairs; the estimated value function is the resulting approximate function that predicts values.