Sarsa and n-step Sarsa

Two Ways to Read Learning Progress

When reinforcement learning is studied with an average-reward objective, the familiar value-function vocabulary needs a parallel version. The source describes differential value functions as the average-reward counterparts of familiar value functions. This is not a cosmetic change: the average-reward setting introduces corresponding differential value functions, differential Bellman equations, and differential temporal-difference errors.

The same parallel idea applies to Sarsa. Semi-gradient Sarsa is useful when the action-value function is represented approximately rather than stored as a fully explicit table. With linear semi-gradient Sarsa and ε-greedy action selection, the reported behavior is especially important: the parameters enter a bounded region near the best solution instead of converging in the usual sense to one fixed point.

The Differential Perspective

Average-reward reinforcement learning does not simply reuse every object from the usual formulation unchanged. Instead, it develops corresponding differential objects. Differential value functions are the value-function counterpart for the average-reward setting, and differential Bellman equations and temporal-difference errors form a parallel set of components. The organizing principle is continuity: the conceptual changes are small enough to form a parallel family, but the average-reward setting still requires its own definitions and algorithms.

One Sample Through an Approximation

Semi-gradient Sarsa becomes relevant when the action-value function is represented approximately rather than treated as a fully explicit table. A sampled transition is used to change the approximated action-value estimate, and the learning process changes the parameter vector that represents that approximation. The word semi-gradient identifies the algorithmic family; the important connection here is that Sarsa is being used with function approximation instead of only with a fully explicit table.

From One Step to n Steps

Sarsa and n-step Sarsa belong to the same broader family, but their backup targets differ in how far into the future the learning information reaches. The one-step view uses the immediate sampled transition and an estimate associated with the next step. An n-step view extends the backup across multiple steps, using rewards and estimates from farther into the future. In the average-reward setting, this parallel family includes differential versions of semi-gradient n-step Sarsa.

ε-Greedy Action Selection

Under ε-greedy action selection, learning chooses between an action currently estimated as best and exploratory actions. Therefore, the sequence of sampled transitions is shaped by both the current estimates and the exploratory part of the policy. The source reports the resulting convergence behavior for linear semi-gradient Sarsa with ε-greedy action selection: the parameters enter a bounded region near the best solution rather than converging in the usual sense.

Bounded Region versus Fixed Point

Entering a bounded region near the best solution means that the parameter values remain within a neighborhood of that solution. Converging in the usual sense would mean approaching one limiting parameter value. These descriptions are not interchangeable: the first permits continued movement inside a bounded neighborhood, while the second describes approach to a single limit.

• Treating a bounded region as identical to a single converged point.

  A neighborhood can contain continuing variation, whereas usual convergence refers to approaching one limiting value.

  Fix: Describe the reported behavior as entering a bounded region near the best solution.

• Assuming average-reward learning reuses every ordinary value and error object unchanged.

  The source identifies differential counterparts for each of these components.

  Fix: Use the differential terminology when discussing the average-reward formulation.

• Ignoring function representation when explaining semi-gradient Sarsa.

  Semi-gradient Sarsa is connected in the source with approximate value-function representation.

  Fix: Explain that the sampled transition changes an approximation represented by parameters.

Practice Check

Key Takeaways

1. Differential value functions are the average-reward counterparts of familiar value functions.
2. Average-reward learning uses a parallel family of differential value functions, Bellman equations, and temporal-difference errors.
3. Semi-gradient Sarsa is connected to function approximation because the action-value function is represented approximately and learning changes its parameter vector.
4. n-step Sarsa extends the backup across multiple steps rather than restricting it to the one-step view.
5. For linear semi-gradient Sarsa with ε-greedy action selection, the reported behavior is entry into a bounded region near the best solution, not convergence in the usual sense to one fixed point.