Concepts / Temporal-Difference Prediction Methods

Temporal-Difference Prediction Methods

Bootstrapping is the process of learning a guess from another guess.

  • Programming

The Learning Pattern

Temporal-difference prediction methods are defined in this material through bootstrapping: learning one guess from another guess. The central question is not simply whether an estimate is correct. Instead, ask how one estimate contributes to improving another estimate.

Bootstrapping means that an estimate already available to the method contributes to the learning of another estimate.

Following the Estimates

To detect bootstrapping, trace the dependency between two estimates. First identify the estimate that the method is trying to update. This is the estimate being learned. Then identify the information that supports that update. If another estimate is part of that support, the method is learning one guess from another guess.

contributessupports learningSupporting estimateavailable guessLearning basiscontributes to updateTarget estimateestimate being learned
How does one guess provide the learning signal used to improve another guess?

A Worked Dependency Trace

Identifying the Two Guesses

Suppose a method is trying to improve Estimate A, and the information used for that improvement includes Estimate B, which is already available to the method. Which estimate is being learned, and which estimate supports its learning?

Identify the target: Estimate A is the estimate being updated, so it is the estimate being learned.

Identify the support: Estimate B contributes to the basis for the update, so it is the supporting estimate.

Name the pattern: Because one estimate contributes to learning another estimate, this is the bootstrapping pattern.

Estimate A is the target estimate, Estimate B is the supporting estimate, and learning A from B is bootstrapping.

This example does not claim that every temporal-difference method uses only two estimates or that the supplied material specifies a particular numerical update. It illustrates the dependency that the material tells you to trace: one estimate is updated, and another estimate contributes to that learning.

Target and Support Roles

The two estimates have different roles. The target estimate is the one the method is trying to improve. The supporting estimate is the other available guess that contributes to the target estimate's learning. Calling both of them simply an estimate can hide the direction of the dependency, so always name their roles separately.

contributes to learningSupporting estimatesupplies supportTarget estimateis updated
Which estimate is updated, and which other estimate supplies the support for that update?

When reading a method description, write down two labels: estimate being updated and estimate providing support. This prevents you from confusing the object of learning with the information that helps produce that learning.

Avoiding Misidentification

  • Treating bootstrapping as merely making an estimate.

    The defining feature is not the existence of an estimate. It is that one estimate contributes to learning another estimate.

    Fix: Trace the dependency and identify both the target estimate and the supporting estimate.

  • Checking only whether an estimate is correct.

    The supplied material says to trace the direction of learning rather than asking whether an estimate is correct.

    Fix: First ask which estimate is being updated, then ask what information contributes to that update.

  • Assuming the supplied comparison gives the detailed advantages of TD methods.

    The supplied material states that TD methods have advantages over both, but it postpones the details and does not list them.

    Fix: State only that the material establishes the existence of advantages; leave the specific reasons for later study.

Check Your Understanding

EASY

A method updates Estimate X. The information contributing to that update includes Estimate Y, which is already available to the method. Identify the target estimate, the supporting estimate, and the bootstrapping relationship.

Hints
  • The target is the estimate being updated.
  • The supporting estimate is the other guess contributing to that update.
  • Describe the relationship as learning one guess from another guess.

What do you think happens?

If a method updates Estimate X using information that includes already available Estimate Y, is this the bootstrapping pattern?

  • Yes, because one estimate contributes to learning another estimate.
  • No, because estimates cannot support other estimates.
  • No, unless the passage supplies the numerical values of both estimates.
Reveal answer

Answer: Yes, because one estimate contributes to learning another estimate.

Bootstrapping is identified by the dependency between the supporting estimate and the target estimate, not by particular numerical values.

What the Passage Establishes

The supplied material identifies bootstrapping as a key characteristic of temporal-difference methods. It also states that TD methods have advantages over Monte Carlo methods and dynamic programming methods. However, this passage does not list those advantages; their detailed explanation belongs to later study.

Key Takeaways

  • Bootstrapping is the process of learning a guess from another guess.
  • The target estimate is the estimate being updated.
  • The supporting estimate is the available estimate that contributes to the target estimate's learning.
  • The clearest way to recognize bootstrapping is to trace the dependency from the supporting estimate to the target estimate.
  • The supplied material says TD methods have advantages over Monte Carlo and dynamic programming methods, but it does not specify what those advantages are.

Related Topics