Introduction to λ-return
Multiple n-step returns can be combined into one composite target.
From One Horizon to Many
An n-step return uses information from a particular backup horizon. Instead of selecting only one horizon, an update can use an average of several n-step returns. The resulting target is called a composite return. A λ-return is introduced as this kind of combined target: one target assembled from multiple simpler n-step targets.
The central change is not the creation of a new individual return. It is the combination of several existing n-step returns into one target.
Weighted Combination
A valid composite return uses positive weights for the component n-step returns. Those weights must add up to 1. Each return therefore contributes part of the final target, while the complete combination remains an average rather than an unscaled sum.
Checking a Composite Target
Suppose a learner combines a short-horizon return, a medium-horizon return, and a long-horizon return.
Assign contributions: Give each of the three n-step returns a positive weight. For illustration, the contributions can be 0.2, 0.3, and 0.5.
Check the total: The three weights add to 1.0, so they form a valid average.
Form the target: Combine the three weighted returns into one composite target. The target contains information from all three backup horizons.
The combination is a valid composite return because every component has a positive weight and the weights add up to 1.
Preserved Error Reduction
The source describes an error reduction property for individual n-step returns and states that a composite return retains a similar property. The reason to care about the weighting rule is that the composite target is an average of the component returns: it does not simply combine them without scale. The averaged target can therefore preserve the useful error-reduction behavior while drawing information from multiple backup horizons.
A useful way to interpret the combination is to treat backup horizon as a choice of how much information to use. A short horizon contributes a nearer-term view, while longer horizons contribute information from farther along the backup. The composite return avoids committing the update to only one of these choices.
Backup Availability
A compound backup cannot be performed as soon as its shortest component is available. It becomes available only when the longest component included in the combination is complete. This timing condition follows from the fact that the composite target depends on every selected n-step return.
If a combination includes a long-horizon return, the shorter returns being complete is not enough to perform the compound backup. The longest selected component controls when the combined target is available.
Connections to Backup Algorithms
Averaging has a broader role than smoothing two targets together. It provides a way to create additional backup algorithms from existing n-step components. The source discusses averaging a one-step return and an infinite-step return as a way to relate temporal-difference and Monte Carlo methods.
The λ-return perspective treats backup methods as points that can be combined through weighted n-step returns, rather than requiring a choice of only one backup horizon.
Common Mistakes
Treating the composite return as just one selected n-step return
A composite return is assembled from multiple n-step returns rather than selecting only one horizon.
Fix:
Identify all included n-step returns and describe how their weighted contributions form one target.Using weights that do not add up to 1
The source requires positive weights that add up to 1 for the combination to be a valid average.
Fix:
Check both conditions: every weight is positive and the complete set of weights sums to 1.Performing the backup when only the shortest return is complete
The compound backup is unavailable until its longest component is complete.
Fix:
Wait until the longest n-step return included in the composite target is complete.Assuming averaging removes the error-reduction property
The source states that a composite return retains an error reduction property similar to individual n-step returns.
Fix:
Recognize the composite return as an averaged target that preserves a similar property.
Check Your Understanding
A proposed composite target uses three n-step returns. The weights are positive, but their total is not 1. The longest return is also incomplete. Which two conditions prevent the compound backup from being treated as a valid, currently available composite backup?
Hints
- Check the rule for a valid average.
- Check which component determines backup availability.
- The proposed target fails both checks. Its weights do not form a valid average because they do not add up to 1, and the backup cannot yet be performed because the longest selected n-step return is incomplete.
Key Takeaways
- A λ-return combines multiple n-step returns into one composite target.
- The component returns must receive positive weights that add up to 1.
- The composite return retains an error reduction property similar to that of individual n-step returns.
- A compound backup is unavailable until the longest included n-step return is complete.
- Weighted combinations can create additional backup algorithms, including connections between one-step and infinite-step methods.