States and Actions in Reinforcement Learning

From Situation to Result

Reinforcement learning tasks can be understood as transitions. An agent begins with the task in its current state, takes an action, and the task moves into a new state. The state reached immediately after that action is called the afterstate.

The Transition Pattern

An afterstate is not the situation before the action and not the action itself. It is the state that follows the action immediately. To identify it reliably, trace the task in order: first describe the current situation, then identify the action by its immediate effect, and finally describe the resulting state.

Separating Before and After

The visual separates three things that are easy to blend together: the original situation, the chosen action, and the state produced by that action. The afterstate is the last of these three.

Queuing as an Afterstate Task

Afterstates are not limited to games. They also apply to queuing tasks, because a queuing action can have a known immediate effect. Examples of such effects include assigning a customer, rejecting a customer, or discarding information.

In a queuing task, identify the action through what it immediately does. If the effect is assignment, the afterstate is the queue-related state immediately after that assignment. If the effect is rejection, the afterstate is the state immediately after rejection. If the effect is discarding information, the afterstate is the state immediately after that information is discarded.

A Queue Transition Walkthrough

Consider a generated queuing example. Begin with the queue in its current state. The selected action assigns a customer. The action is named by that immediate effect, not by a later consequence. The afterstate is the queue state immediately after the customer has been assigned.

Mistakes in State Tracing

• Calling the current state the afterstate

  The afterstate is reached immediately after the action, while the current state exists before the action.

  Fix: Keep the before-action situation as the current state and label the immediately following situation as the afterstate.

• Treating the action and the afterstate as the same thing

  Assignment is the immediate effect that defines the action; the afterstate is the state produced by that effect.

  Fix: Name the action first, then describe the task immediately after that action.

• Assuming afterstates belong only to games

  The concept applies whenever an action produces a known immediate result, including assignment, rejection, or discarding information in queuing tasks.

  Fix: Apply the same transition test to any task: current state, immediate action effect, resulting afterstate.

• Focusing on a later result instead of the immediate result

  The definition is tied to the state reached immediately after the action.

  Fix: Stop the trace at the first state produced by the chosen effect.

Practice the Transition

The Essential Distinction

1. An afterstate is the state reached immediately after an action.
2. The current state describes the task before the action; the afterstate describes the task directly afterward.
3. To find an afterstate, trace the immediate effect of the chosen action.
4. Queuing actions can be defined by immediate effects such as assignment, rejection, or discarding information.
5. The state-to-action-to-afterstate pattern applies beyond games whenever an action produces a known immediate result.