Average Reward Formulation and Semi-gradient Methods
Episodic n-step semi-gradient Sarsa is based on the forward Sarsa(λ) algorithm of van Seijen (2017).
From Episodic Learning to Continuing Control
This topic connects several ideas rather than presenting a complete implementation. The source identifies episodic n-step semi-gradient Sarsa as being based on the forward Sarsa(λ) algorithm of van Seijen (2017). It then turns to continuing control, where the objective must be reformulated around average reward per time step. The key skill is to follow these relationships without adding update equations or pseudocode that the source does not provide.
Two Control Objectives
The source distinguishes episodic control from continuing control. In the episodic case, parameterized function approximation and semi-gradient descent extend naturally to control. Continuing control is different because the agent continues acting rather than working toward the end of an episode. For that setting, the formulation is changed so that the objective is to maximize average reward per time step.
The important change is not merely a different name for the same objective. The source presents average reward per time step as the formulation needed for continuing control, whereas the episodic discussion concerns the discounted formulation and its extension with function approximation.
Why Approximation Changes Control
The source identifies a limitation of carrying the discounted formulation into approximate control: most policies cannot be represented by a value function in the approximate case. Control still requires policies to be compared and ranked, but a value-function representation is unavailable for most of those policies. This is why the source introduces a different role for the average-reward formulation.
The Scalar η(π)
η(π) is the scalar average reward associated with policy π. In the source's formulation, this scalar provides a common quantity for ranking arbitrary policies.
The distinction between representation and ranking is central. A value function is a representation of values, and the source says that most policies cannot be represented in that way when approximations are present. η(π) serves a different purpose: it supplies one scalar by which policies can be ordered. The policy-ranking role does not require every policy to have its own usable approximate value-function representation.
Ranking Two Policies with η(π)
Suppose two hypothetical continuing policies, Policy A and Policy B, are evaluated by their scalar average rewards. Explain how the policies can be ranked without constructing a usable approximate value-function representation for both.
Assign the policy quantities: Associate η(πA) with Policy A and η(πB) with Policy B. This is a conceptual illustration, not a numerical calculation from the source.
Compare the scalars: Compare η(πA) and η(πB). The policy with the larger scalar average reward receives the higher ranking under the average-reward formulation.
Separate the roles: The comparison ranks the policies through their scalar average rewards. It does not claim that each policy has a separately available approximate value function.
η(π) changes the comparison task from requiring value-function representation for most policies to ordering policies by a common scalar average reward.
Algorithmic Relationships
The source states that episodic n-step semi-gradient Sarsa is based on forward Sarsa(λ), associated here with van Seijen (2017). Separately, it characterizes the algorithm described in the section as the on-policy analog of R-learning, which was introduced by Schwartz in 1993. These are relationships and historical attributions, not a complete derivation of the algorithms.
The average-reward formulation is not presented as belonging exclusively to one area. The source says it has been discussed in both dynamic programming and reinforcement learning. It also mentions an access-control queuing example associated with Carlström and Nordström (1997), but does not provide the queue's states, actions, rewards, or calculations.
Evidence and Implementation Boundaries
| Established by the source | Not provided by the source |
|---|---|
| Episodic n-step semi-gradient Sarsa is based on forward Sarsa(λ). | A pseudocode listing for the algorithm. |
| The described algorithm is identified as the on-policy analog of R-learning. | Specific update equations. |
| Average-reward methods are discussed in dynamic programming and reinforcement learning. | A numerical execution trace. |
| η(π) can rank arbitrary policies. | Numerical policy data or a calculated average reward. |
| Continuing control is formulated around average reward per time step. | Details of a particular implementation. |
Use this boundary to avoid turning bibliographical relationships into unsupported implementation claims.
Treating the source's algorithm relationships as a complete algorithm specification.
The source pack explicitly says that pseudocode, update equations, and a step-by-step execution trace are not provided.
Fix:
State the relationship that is given, then label any implementation detail as unspecified.Confusing value representation with policy ranking.
The source says most policies cannot be represented by a value function in the approximate case, while η(π) can still provide a scalar for ranking.
Fix:
Explain that a value function represents values, whereas η(π) supplies a common scalar for ordering policies.Using the access-control queuing example as if its numerical details were supplied.
The source only identifies the example's bibliographical association with Carlström and Nordström (1997).
Fix:
Treat it as a historical reference unless independent details are supplied.Saying that the average-reward formulation is only a reinforcement-learning idea.
The source says the formulation has been discussed in both dynamic programming and reinforcement learning.
Fix:
Mention both settings when describing the formulation's scope.
Check Your Understanding
Explain, in your own words, why the average-reward formulation is useful for approximate continuing control. Your answer should mention the limitation involving value-function representation and the role of η(π) in ranking policies.
Hints
- Start by stating what changes when control is continuing.
- Explain why most policies create a problem for approximate value-function representation.
- Finish by describing what the scalar η(π) makes possible.
Classify each statement as established by the source or unspecified by the source: the algorithm is an on-policy analog of R-learning; the exact update equation; the average-reward formulation appears in dynamic programming and reinforcement learning; the numerical average reward of the access-control queuing example.
Hints
- Historical and conceptual relationships are stated directly.
- Look for details that would require pseudocode, equations, or numerical data.
Key Takeaways
- Episodic n-step semi-gradient Sarsa is based on forward Sarsa(λ), associated in the source with van Seijen (2017).
- The described algorithm is characterized as the on-policy analog of R-learning, introduced by Schwartz in 1993.
- Continuing control uses a formulation based on maximizing average reward per time step.
- Approximate control creates a representation problem because most policies cannot be represented by a value function.
- The scalar η(π) provides a way to rank arbitrary policies without making policy ranking identical to value-function representation.
- The source establishes conceptual and historical relationships but does not provide pseudocode, update equations, or numerical execution details.
Key Takeaways
- Continuing control requires an average-reward-per-time-step formulation.
- Approximate control cannot simply reuse the discounted formulation because most policies cannot be represented by a value function.
- η(π) supplies a scalar for ranking policies, separating policy comparison from value-function representation.
- The source links episodic n-step semi-gradient Sarsa to forward Sarsa(λ) and the described algorithm to R-learning's on-policy counterpart.
- The discussion is bibliographical and does not specify implementation equations or execution steps.