Episodic Tasks in Reinforcement Learning

The Question Behind the Distribution

When reinforcement learning evaluates a policy, it does not necessarily treat every state as equally important. The useful question is: under this policy, which states are encountered, and how often? The on-policy distribution answers this question by describing how state probability is distributed under the policy.

In an episodic task, this distribution depends on the complete way an episode unfolds. The episode-start rule determines where visits begin, and policy-following transitions determine where later visits occur.

Starting States Shape Later Visits

Initial-state selection is part of the distribution calculation in an episodic task. Two settings can use the same policy and the same transition behavior but still produce different expected visits if their episode-start rules differ.

A state that receives more episode starts can accumulate more expected visits before normalization. A state that is rarely selected initially may still receive many visits later if policy-following transitions lead into it. Therefore, initial-state choices affect not only the first state in an episode, but potentially the entire on-policy distribution.

Tracing Visits Through an Episode

A useful way to trace the calculation is to follow the source of every visit. Some visits occur immediately because the episode-start rule places an episode in a state. Other visits occur later because policy-controlled transitions move the agent from states already being visited into additional states.

1. Identify how episodes begin and which states receive those starts.
2. Follow the transitions produced while the policy is followed.
3. Add the contributions from initial visits and later transition-based visits.
4. Obtain an expected visit count for each state.
5. Normalize those expected counts to obtain probabilities that sum to one.

This order matters. First determine how many visits each state is expected to receive. Only after that should you convert those counts into a probability distribution.

Episodes and Their Objective

Organizing a task into episodes gives the agent repeated sequences with defined beginnings and endings. Each episode supplies a path through the environment: it starts according to the initial-state rule, continues through interactions under the policy, and ends at the task's specified endpoint.

The objective of an episodic task is to maximize the expected total reward. Rewards guide the agent toward the desired objective, so the reward system must communicate what achievement is wanted.

A Worked Visit Calculation

The same separation applies when initial-state choices are involved. First include the visits supplied by episode starts and the additional visits supplied by policy-following transitions. Then normalize the resulting expected counts.

Mistakes in Distribution Reasoning

• Treating every state as equally important

  The on-policy distribution describes which states are encountered, and how often, under a particular policy.

  Fix: Trace episode starts and policy-following transitions to determine expected visits.

• Ignoring the initial-state distribution

  Initial-state selection is part of the distribution calculation for episodic tasks.

  Fix: Include the episode-start rule before accounting for later transitions.

• Stopping at expected visit counts

  Expected visits are counts. They do not yet form probabilities that sum to one.

  Fix: Normalize the expected visit counts.

• Mixing the visit and normalization stages

  This makes it difficult to tell whether an error came from modeling visits or from converting counts into probabilities.

  Fix: Keep the calculation in two layers: expected visits first, normalization second.

• Assuming a weak reward system will still clearly guide learning

  Ineffective reward systems can hinder learning and may not communicate the desired objective effectively.

  Fix: Investigate whether the reward system clearly guides the agent toward the task objective.

Practice the Two-Layer Method

A strong answer should distinguish three ideas: where visits begin, how later visits arise, and how expected counts become probabilities. It should also recognize that changing the initial-state rule can change the final on-policy distribution even when the policy and transition behavior remain the same.

Key Takeaways

1. The on-policy distribution describes how state probability is distributed under a policy. In episodic tasks, its calculation includes both the initial-state distribution and the transitions that occur while following the policy. Expected visits must be calculated before normalization, because normalization converts visit counts into probabilities that sum to one. Episodes have defined starts and ends, their objective is to maximize expected total reward, and rewards guide the agent toward that objective.