Concepts / Action-value update rule

Action-value update rule

The n-step target is a quantity used by an action-value update rule.

  • Programming

Two quantities to track

The key distinction is between the n-step target and the action-value update rule. The n-step target is a quantity used by an action-value update rule. It supplies a learning quantity, while the update rule determines how that quantity is used to update an action-value.

Target-to-update flow

First, an n-step target is obtained. That target is then supplied to an action-value update rule. The action-value estimate is the quantity that is updated as a result. In other words, the target is an input to the update process, not the name of the process itself.

inputupdatesn-step targetlearning quantityaction-value updateruleuses the targetaction-value estimateupdated quantity
How does the n-step target flow into the action-value update rule, and what changes as a result?

Separating the two roles

Suppose an n-step target has already been obtained for an action-value estimate.

Identify the target: The n-step target is the learning quantity that has been computed.

Identify the rule: The action-value update rule is the mechanism that uses the target.

Identify the changed quantity: The action-value estimate is the quantity updated by the rule. The target itself is not renamed as the update rule.

The target is an input to the action-value update rule, and the action-value estimate is what the rule updates.

What changes after the target exists

Once the n-step target has been obtained, the important question is what happens next. The target does not by itself describe the complete learning operation. It is used with the usual action-value update rule from n-step Sarsa. This is why a description of a target must be followed by a description of the update rule that consumes it.

used by update rulen-step targetcomputed quantityn-step targetinput to ruleaction-valueestimatecurrent estimateaction-value estimateupdated estimate
Which quantity is computed as the target, which quantity is changed, and how are they related?

When reading or explaining this formulation, name the target and the update separately. A clear explanation should say both what quantity was obtained and that the quantity is then used by the action-value update rule.

Tree Backup and n-step Sarsa

At the level stated in the source, the Tree Backup Algorithm uses its target with the usual action-value update rule from n-step Sarsa. The shared point is the action-value update rule. The target is the quantity supplied to that rule. The source does not provide the internal construction details needed to describe the target paths more specifically, so the safe distinction is that target construction and target use are separate parts of the explanation.

used withassociated withTree Backup targettarget quantityn-step targettarget quantityusual action-valueupdate ruleused after target isobtained
Which relationship between Tree Backup and n-step Sarsa is established here?

The zero-factor convention

A product of zero factors is defined as 1. This convention applies when a product in the target formulation contains no factors. The empty product therefore contributes 1 rather than 0.

An empty product

A boundary case calls for a product, but there are zero factors to multiply.

Count the factors: There are no factors in the product.

Apply the convention: A product of zero factors is defined as 1.

Use the result: The boundary case contributes 1 wherever that empty product appears in the target or update formulation.

The empty product has value 1.

Common interpretation mistakes

  • Calling the n-step target the update rule.

    The source distinguishes the target from the action-value update rule. The target is a quantity used by the rule.

    Fix: Say that the target is supplied to the action-value update rule, which then updates an action-value estimate.

  • Describing only the target and omitting what happens afterward.

    The important question includes what happens after the target has been obtained.

    Fix: Continue by identifying the usual action-value update rule that uses the target.

  • Treating an empty product as zero.

    The stated convention defines a product of zero factors as 1.

    Fix: Use 1 for the empty product.

  • Claiming more detail about Tree Backup and n-step Sarsa than the source states.

    The source establishes the use of the usual action-value update rule but does not supply those internal details.

    Fix: State only that the Tree Backup target is used with the usual action-value update rule from n-step Sarsa.

Check your understanding

EASY

Explain the sequence in your own words: an n-step target is obtained, and then an action-value update rule is applied. Identify which quantity is the input and which quantity is updated.

Hints
  • The target supplies a learning quantity.
  • The action-value estimate is the quantity changed by the rule.
  • Do not use target and update rule as synonyms.
MEDIUM

In a boundary case with a product containing zero factors, what value should the product contribute? Then state how the Tree Backup target is related to the usual action-value update rule from n-step Sarsa.

Hints
  • Use the stated empty-product convention.
  • The relationship concerns how the target is used after it is obtained.

Essential distinctions

  1. The n-step target is a quantity used by an action-value update rule.
  2. The target is not the same thing as the update rule.
  3. The update rule uses the target to update an action-value estimate.
  4. In the stated Tree Backup formulation, the target is used with the usual action-value update rule from n-step Sarsa.
  5. A product of zero factors is defined as 1.

Key Takeaways

  • Separate target construction from action-value updating.
  • Treat the n-step target as an input to the update rule, not as the update itself.
  • The stated Tree Backup relationship is that its target is used with the usual action-value update rule from n-step Sarsa.
  • Use 1 for a product containing zero factors.

Related Topics