Monte Carlo backup
The λ-return is a normalized weighted average of n-step backups.
Why combine look-ahead lengths
An n-step backup estimates what a state is worth by looking ahead a chosen number of steps. A one-step backup looks ahead one step, a two-step backup looks ahead two steps, and later backups look farther ahead. The λ-return does not choose only one of these possibilities. It combines the available n-step returns into one return, giving each look-ahead length a different weight.
The central idea is a weighted average. The first return receives weight 1 − λ. The next return receives (1 − λ)λ, and the third receives (1 − λ)λ². Each additional step multiplies the previous weight by λ, so the contribution fades as the look-ahead becomes longer.
The geometric weight pattern
The λ-return is a normalized weighted average of n-step backups. Its weights follow the sequence 1 − λ, (1 − λ)λ, (1 − λ)λ², and so on.
Weight for the first return = 1 − λ
Weight for the second return = (1 − λ)λ
Weight for the third return = (1 − λ)λ²
Weight for a later return = (1 − λ)λ^(n − 1)The factor λ controls how quickly the weights decrease. To obtain the next weight, multiply the current weight by λ. If λ is small, the weights shrink quickly and the earliest return dominates. If λ is larger, later returns retain more influence.
A numerical weighting example
Weights when λ equals 0.5
Calculate the first four weights when λ = 0.5.
One-step return: The first weight is 1 − λ, so it is 1 − 0.5 = 0.5.
Two-step return: Multiply the first weight by λ: 0.5 × 0.5 = 0.25.
Three-step return: Multiply the second weight by λ: 0.25 × 0.5 = 0.125.
Four-step return: Multiply the third weight by λ: 0.125 × 0.5 = 0.0625.
The first four weights are 0.5, 0.25, 0.125, and 0.0625. Each weight is one-half of the preceding weight.
The numbers above show the weighting rule, not particular values for the n-step returns themselves. To form the λ-return, each n-step return is multiplied by its corresponding weight, and the weighted contributions are combined. The weighting rule determines how strongly each look-ahead length contributes.
The λ extremes
Changing λ changes where the backup gets its information. At λ = 0, the first weight is 1 and the later weights are 0. The combined backup therefore keeps only the one-step TD backup. As λ moves toward 1, later n-step returns receive more influence. At λ = 1, the overall backup reduces to its last component, the Monte Carlo backup.
| Value of λ | Weight distribution | Backup behavior |
|---|---|---|
| 0 | All weight on the first return | One-step TD backup |
| Between 0 and 1 | Weight begins with the first return and fades across later returns | Combination of n-step backups used by TD(λ) |
| 1 | The overall backup reduces to its last component | Monte Carlo backup |
When reasoning about a λ-return, first identify λ, then calculate the first weight, and generate later weights by multiplying by λ repeatedly. Finally, check the endpoint behavior: λ = 0 should leave only the one-step backup, while λ = 1 should leave the Monte Carlo backup.
Common weighting mistakes
Using λ itself as the first weight
The first return receives 1 − λ, not λ. The later weights are generated by multiplying that first weight by λ.
Fix:
Calculate the first weight as 1 − λ, then multiply by λ for each additional return.Adding λ instead of multiplying by λ
The weighting pattern decreases by a factor of λ with each additional step.
Fix:
Generate the sequence as 1 − λ, then (1 − λ)λ, then (1 − λ)λ².Treating every n-step return as equally important
The λ-return is a weighted average, and the weights fade as the number of steps increases.
Fix:
Assign each return its position-dependent weight.Assuming λ = 0 and λ = 1 have the same backup behavior
At λ = 0, only the one-step TD backup remains. At λ = 1, the overall backup reduces to its last component, the Monte Carlo backup.
Fix:
Use the endpoint meanings as a check on your interpretation.
Practice the calculation
Let λ = 0.25. Calculate the weights for the one-step, two-step, and three-step returns. Then explain which return receives the largest weight and why.
Hints
- Begin with 1 − λ.
- Multiply each weight by λ to obtain the next weight.
- Compare the resulting values in order from the one-step return to the three-step return.
Checking the practice result
For λ = 0.25, calculate the first three weights.
One-step return: 1 − 0.25 = 0.75.
Two-step return: 0.75 × 0.25 = 0.1875.
Three-step return: 0.1875 × 0.25 = 0.046875.
The weights are 0.75, 0.1875, and 0.046875. The one-step return receives the largest weight because it is the first return, whose weight is 1 − λ.
Key takeaways
- The λ-return combines available n-step backups into one normalized weighted average.
- The weights begin with 1 − λ and then follow (1 − λ)λ, (1 − λ)λ², and so on.
- Each additional step multiplies the previous weight by λ, so the contribution fades as the look-ahead grows.
- At λ = 0, TD(λ) uses only the one-step TD backup; at λ = 1, the overall backup reduces to the Monte Carlo backup.
- After a terminal state, later n-step returns are all equal to G_t.
Key Takeaways
- The λ-return is a normalized weighted average of n-step returns.
- Its weights are 1 − λ, (1 − λ)λ, (1 − λ)λ², and later terms formed by repeated multiplication by λ.
- Smaller λ emphasizes short look-ahead backups, while larger λ gives later returns more influence.
- TD(λ) connects one-step TD at λ = 0 with a Monte Carlo backup at λ = 1.
- Once a terminal state is reached, extending the look-ahead produces the same return G_t.