Advantages of Temporal Difference Prediction Methods
Bootstrapping is the process of learning a guess from another guess.
The Central Learning Pattern
Temporal-difference methods are characterized in the supplied material by bootstrapping. Bootstrapping means learning a guess from another guess. Instead of relying only on a final, independently established answer, the method allows one estimate to contribute to the learning of another estimate.
Bootstrapping is the process of learning a guess from another guess.
Following the Estimates
To detect bootstrapping, trace the direction of learning. First identify the estimate that the method is trying to improve. This is the target estimate: the estimate being updated. Then identify the information that contributes to that update. If another estimate is part of that basis, the method is learning one guess from another guess.
The supporting estimate and the target estimate have different roles. The target estimate is the one being learned or improved. The supporting estimate supplies part of the basis for that learning.
A Trace of Bootstrapping
Tracing Two Estimates
A method is improving Estimate A. The update uses Estimate B, which is already available to the method. Does this describe bootstrapping?
Find the target: Estimate A is the target because it is the estimate the method is trying to improve.
Find the support: Estimate B is the supporting estimate because information from it contributes to the update of Estimate A.
Trace the dependency: The learning direction runs from Estimate B to Estimate A. One estimate is therefore contributing to the learning of another estimate.
Name the pattern: Because the method learns one guess from another guess, the pattern is bootstrapping.
Yes. Estimate A is the target estimate, Estimate B is the supporting estimate, and the dependency between them is bootstrapping.
The example does not require Estimate B to be correct. Bootstrapping is identified by the role Estimate B plays in the learning process: it is another estimate that contributes to the update of Estimate A.
What the Comparison Establishes
| Method named in the material | What the supplied material establishes |
|---|---|
| Temporal-difference methods | Bootstrapping is identified as a key characteristic. |
| Monte Carlo methods | The material says TD methods have advantages over Monte Carlo methods, but does not list those advantages. |
| Dynamic programming methods | The material says TD methods have advantages over dynamic programming methods, but does not list those advantages. |
Mistakes in Tracing Learning
Looking only at whether an estimate is correct
The defining issue is the dependency between the supporting estimate and the target estimate, not whether the supporting estimate is correct.
Fix:
Identify the estimate being updated and then ask whether another estimate contributes to that update.Confusing the target estimate with the supporting estimate
The target is the estimate being updated. The supporting estimate supplies part of the basis for that update.
Fix:
Trace the direction of learning from the supporting estimate to the target estimate.Adding unprovided advantages to the comparison with other methods
The material states that TD methods have advantages over both, but postpones the details.
Fix:
State only that the advantages are identified as existing; leave their detailed explanation for later study.
Practice the Dependency Test
A method updates Estimate X. The information used in that update includes Estimate Y, which is another estimate already available to the method. Identify the target estimate, the supporting estimate, and the learning pattern.
Hints
- Start with the estimate that is being updated.
- Then identify the other estimate that contributes to the update.
- Use the definition of learning a guess from another guess.
Practice Solution
A method updates Estimate X using information that includes Estimate Y.
Target estimate: Estimate X is the target because it is the estimate being updated.
Supporting estimate: Estimate Y is the supporting estimate because it contributes to the update.
Learning pattern: The method is learning one guess from another guess, which is bootstrapping.
Estimate X is the target, Estimate Y is the support, and the dependency describes bootstrapping.
Key Takeaways
- Bootstrapping is learning a guess from another guess.
- In TD methods, one estimate can contribute to the learning of another estimate.
- The target estimate is the estimate being updated; the supporting estimate provides part of the basis for that update.
- The clearest way to identify bootstrapping is to trace the dependency between the supporting estimate and the target estimate.
- The supplied material says that TD methods have advantages over Monte Carlo and dynamic programming methods, but it does not provide the details of those advantages.
Key Takeaways
- Bootstrapping means learning a guess from another guess.
- A TD method can use one estimate to support the learning of another estimate.
- To recognize the pattern, identify the estimate being updated and trace which other estimate contributes to that update.
- The supplied material establishes that TD methods have advantages over Monte Carlo and dynamic programming methods, while leaving the specific advantages for later study.