Discounted Formulation

The Continuing-Control Problem

Some control problems are episodic: they have a natural episode structure, and parameterized function approximation together with semi-gradient descent extends naturally to control in that setting. Continuing problems are different. They do not use the same episodic framing, so they require a new problem formulation based on maximizing the average reward per time step.

The important shift is the objective. In continuing control, the task is formulated around the reward obtained on average at each time step. This gives the learner a way to evaluate ongoing behavior rather than relying on an episodic control formulation.

Two Formulations for Control

Where Approximation Breaks the Argument

The discounted formulation has a limitation when control uses approximation. In the approximate case, most policies cannot be represented by a value function. That creates a direct problem: control still needs to compare policies, but a value-function representation is unavailable for most of the policies being considered.

Ranking Policies with η(π)

The key point is not that η(π) describes every internal detail of a policy. Its role is to provide one scalar average reward for that policy. Those scalar quantities can then be used to order arbitrary policies, even when most of the policies cannot be represented by value functions.

Representation and Ranking

Value-function representation and policy ranking are related but different tasks. A value function is a representation issue: can the policy be represented in that form? η(π) addresses the ranking issue: once policies are associated with scalar average rewards, how can they be ordered? In the approximate case, the first task may fail for most policies, while the second task still has to be solved.

Common Misunderstandings

• Treating continuing control as if it were simply the episodic case extended indefinitely.

  The source states that continuing cases require a new problem formulation based on maximizing average reward per time step.

  Fix: Start by identifying the setting. For continuing control, use the average reward formulation as the objective described by the source.

• Assuming the discounted formulation can always be carried over when approximation is used.

  In the approximate case, most policies cannot be represented by a value function.

  Fix: Recognize the limitation and use η(π) as a scalar quantity for ranking arbitrary policies.

• Treating η(π) as another name for a value-function representation.

  η(π) is a scalar average reward used to rank policies; the source distinguishes this role from representing a policy with a value function.

  Fix: Separate representation from ranking. A policy may lack the relevant value-function representation while still being assigned an η(π) for comparison.

• Expecting the concept example to require numerical reward data.

  The source provides no reward sequence or numerical policy data and uses the example only to show the role of η(π).

  Fix: Focus on the structural idea: η(π) supplies a common scalar with which arbitrary policies can be ordered.

Apply the Distinction

Key Takeaways

1. Continuing control requires a formulation based on maximizing average reward per time step.
2. The discounted formulation extends naturally with parameterized function approximation and semi-gradient descent in the episodic case, but it cannot be carried over to approximate control in the continuing case.
3. In approximate control, most policies cannot be represented by a value function.
4. The scalar η(π) supplies a common average-reward quantity for ranking arbitrary policies.
5. Value-function representation and policy ranking are separate roles: the first may be unavailable for most policies, while η(π) still supports comparison.