Nonstationary Problems and Weighted Averages

When the Past Becomes Less Useful

In a nonstationary problem, the reward situation can change over time. Because of that, an old reward may not be as useful for estimating the present as a recent reward. An exponential recency-weighted average addresses this by giving recent rewards more influence and older rewards progressively less influence.

Recent Rewards Lead the Estimate

The method assigns a weight to every observed reward. The most recent reward has the greatest influence among the observed rewards. A reward one step farther in the past receives one additional factor of 1 − α. A reward another step farther back receives another factor of 1 − α. Since 1 − α is less than 1 in this weighting pattern, repeated multiplication makes the weights smaller.

The diagram shows a pattern rather than a particular numerical calculation. When a new reward arrives, it receives the basic factor α. Previously observed rewards move farther into the past, so their coefficients gain additional factors of 1 − α.

The Weighting Formula

weight of Rᵢ = α(1 − α)ⁿ⁻ⁱ

The initial estimate Q₁ also remains part of the weighted average. Its weight after n rewards is (1 − α)ⁿ. Thus, the current estimate contains a contribution from the initial estimate as well as contributions from the observed rewards.

Qₙ₊₁ = (1 − α)ⁿQ₁ + Σ from i = 1 to n of α(1 − α)ⁿ⁻ⁱRᵢ

All of these weights add to 1: (1 − α)ⁿ + Σ from i = 1 to n of α(1 − α)ⁿ⁻ⁱ = 1. That is why the result is a weighted average: the total contribution is distributed between the initial estimate and the observed rewards.

A Weight Calculation

The Role of α

The constant step-size parameter α determines the basic weight assigned to a newly observed reward. In the weighting formula, the most recent reward has coefficient α. The previous information is reduced through factors of 1 − α. Therefore, α sets the balance between the new reward's direct contribution and the contribution retained from earlier information.

Why Recency Helps Adaptation

When the reward distribution changes, an estimate based equally on all past rewards may allow distant history to retain too much influence. Exponential recency weighting instead emphasizes rewards from the more recent situation while preserving progressively smaller contributions from earlier observations. This makes the weighting pattern suited to nonstationary problems.

The method adapts through repeated updates: every newly observed reward receives the direct factor α, while every older reward acquires another factor of 1 − α. The estimate therefore shifts toward recent evidence without eliminating the past immediately.

Common Weighting Mistakes

• Giving every reward the same weight

  The method is specifically designed to give greater weight to recent rewards and smaller weight to older rewards.

  Fix: Count the number of intervening rewards and use α(1 − α)ⁿ⁻ⁱ.

• Forgetting the initial estimate

  The initial estimate has coefficient (1 − α)ⁿ and remains part of the weighted average.

  Fix: Include the initial-estimate term before checking that all coefficients add to 1.

• Using the number of elapsed time steps instead of intervening rewards

  The exponent n − i counts the intervening rewards.

  Fix: Start at the index of Rᵢ, identify the current reward index n, and calculate n − i.

• Subtracting a fixed amount for each older reward

  The decrease is exponential: each additional intervening reward multiplies the previous weight by 1 − α.

  Fix: Apply another factor of 1 − α for every additional intervening reward.

Check Your Understanding

Key Takeaways

1. A nonstationary problem can make old rewards less useful for estimating the present.
2. The weight of reward Rᵢ is α(1 − α)ⁿ⁻ⁱ, where n − i is the number of intervening rewards.
3. Each additional intervening reward multiplies an older reward's weight by 1 − α, producing exponential decay.
4. The initial estimate Q₁ has weight (1 − α)ⁿ.
5. All coefficients add to 1, so the estimate is a weighted average that emphasizes recent rewards.