Returns in Reinforcement Learning

Why One Reward Is Not Enough

A reinforcement-learning agent receives rewards over time rather than usually receiving one single score. To describe what the agent is trying to optimize, we combine those rewards into a longer-term quantity called the return. The agent's objective is to maximize expected return, not merely one isolated reward.

Tracing Rewards After Time t

Choose a time step called t. The rewards received after that point form an ordered sequence. The first reward after t is R t+1, followed by R t+2, R t+3, and so on. The ordering matters because each reward belongs to a particular point after t.

The return G t is defined from this ordered reward sequence by a specified function. It is not based on an unlabelled collection of rewards: the sequence begins after the selected time step and preserves the position of each reward.

Adding a Finite Reward Sequence

The simplest return function adds the rewards in the sequence. For one particular sequence, this produces one actual return.

When Interaction Has No Endpoint

Some reinforcement-learning problems do not have a natural point at which one interaction ends and another begins. The agent and environment continue interacting without limit. These interactions are called continuing tasks.

This differs from an interaction that breaks naturally into identifiable episodes. In an episodic interaction, an episode provides a recognizable unit that ends before another interaction begins. A continuing task has no such natural division and instead keeps going.

The Infinite-Horizon Problem

Because a continuing task has no final interaction step, its final time step is represented as T = ∞. The standard return formulation can therefore produce an infinite return.

Actual Return and Expected Return

For one particular realized reward sequence, applying the return definition produces one actual return. The expected return is different: it is the expected value of the return. Reinforcement learning focuses on the expected quantity because the agent seeks to maximize expected return rather than merely one isolated outcome.

Common Reasoning Errors

• Treating the return as only the next reward

  The return is defined from the sequence of rewards after t, not necessarily from only the first reward.

  Fix: First identify R t+1, R t+2, R t+3, and the remaining rewards included by the chosen return definition.

• Ignoring reward order

  Each reward belongs to a particular point after t, so the sequence and its indexing matter.

  Fix: Write the rewards in order, beginning with R t+1.

• Assuming every interaction has a final time step

  Continuing tasks do not naturally divide into identifiable episodes and continue without limit, with T represented as ∞.

  Fix: Check whether the interaction has a natural endpoint before treating its reward sequence as finite.

• Confusing an actual return with expected return

  The value 8 is the actual return for one particular sequence. Expected return is the expected value of the return.

  Fix: Use actual return for one realized sequence and expected return for the quantity the agent seeks to maximize.

Practice the Distinction

Key Takeaways

1. A return combines rewards received over time into one quantity using a specified function.
2. After time t, the reward sequence begins with R t+1, R t+2, R t+3, and continues in order.
3. The simplest return function adds the rewards in the sequence.
4. Continuing tasks do not naturally divide into identifiable episodes and have no finite final time step; T is represented as ∞.
5. One realized reward sequence produces an actual return, while the agent seeks to maximize expected return.