State and Action Representations in Reinforcement Learning
An afterstate describes the task immediately after an action.
From Decision to New Situation
In reinforcement learning, selecting an action is not the end of a decision. The action produces a new situation. An afterstate is the task description immediately after that action has been applied. Learning to recognize this post-action situation gives you another way to represent the same task.
An afterstate is the state produced immediately after an action. It is different from the original state that existed before the decision.
Tracing the Afterstate
Keep three items separate when tracing a transition: the original state, the action, and the state produced by that action. The third item is the afterstate. This separation prevents the mistake of treating the afterstate as merely another name for the original state.
A simple transition trace
Trace a hypothetical task in which the original state is labeled State A, the selected action is labeled Action B, and the action produces State C.
Identify the original state: State A describes the situation before the decision.
Identify the action: Action B is the event selected while the task is in State A.
Identify the produced state: State C is the situation that exists immediately after Action B. In this trace, State C is the afterstate.
Keep the roles distinct: State A is not the afterstate. Action B connects the original state to the afterstate, State C.
Original state: State A. Action: Action B. Afterstate: State C.
Reframing the Control Problem
An afterstate reformulation changes what receives primary attention. In the ordinary description, the focus begins with the situation before the decision and the action taken from it. In the reformulated description, the focus is the situation produced by that action. The action remains part of the task; it is still the event that produces the new state.
Reformulating a task around afterstates means starting with the task's ordinary state-and-action description, tracing the action to the state it produces, and making that post-action state the central description.
Jack's Car Rental is the task used in the source exercise for applying this idea. A suitable analysis begins by identifying the situation before the decision, identifying the action, and then describing the situation that exists after the action. That post-action description is the afterstate formulation.
Why Convergence Can Improve
The practical reason to consider an afterstate representation is that it may speed convergence. In this context, the question is whether learning settles more quickly when the task is described through the situation produced by an action. This is a potential advantage, not a guaranteed result.
For Jack's Car Rental, the useful comparison is between two descriptions of the same task: the ordinary state-and-action description and the description centered on the state produced by the action. The key evaluation question is whether the afterstate description helps the learning process settle more quickly.
Original State Versus Afterstate
| Item | When it is considered | Role in the transition |
|---|---|---|
| Original state | Before the action | Describes the situation from which the decision is made |
| Action | At the decision | Produces the new situation |
| Afterstate | Immediately after the action | Describes the situation brought about by the action |
The three items should be kept distinct when tracing a task transition.
The original state and afterstate are related by an action, but they occupy different points in the transition. The original state comes first; the afterstate is the result produced by the action.
Common Representation Mistakes
Treating the afterstate as another name for the original state
The afterstate is defined by its position immediately after the action, so it cannot be identified with the pre-action situation.
Fix:
Write the three items separately: original state, action, and state produced by the action.Removing the action from the reformulated task
An afterstate reformulation does not remove actions. The action remains the event that produces the new state.
Fix:
Keep the action in the transition, even when the afterstate becomes the main object of description.Assuming faster convergence is automatic
The potential benefit depends on the specific task.
Fix:
Compare the ordinary and afterstate descriptions for the task being studied and evaluate whether learning settles more quickly.
When analyzing a new task, first write the original state, then the action, and finally the state produced by that action. Only after those three items are clear should you decide whether the afterstate should become the central representation.
Check Your Understanding
A task is described using an original state, an action, and the state produced by that action. Explain which of the three is the afterstate, how the action connects the other two, and why an afterstate reformulation might be considered for a particular task.
Hints
- The afterstate is identified by when it occurs.
- Keep the original state, action, and produced state separate.
- Mention that faster convergence is a possible task-specific advantage, not a guarantee.
What do you think happens?
Suppose State A is followed by Action B, which produces State C. Which state is the afterstate?
Reveal answer
Answer: State C
The afterstate is the state that exists immediately after the action has been applied.
Key Takeaways
- An afterstate is the situation immediately after an action has been applied.
- A transition should be traced using three separate items: the original state, the action, and the produced afterstate.
- An afterstate reformulation changes the focus of the task description but does not remove actions.
- The possible advantage is faster convergence, but whether convergence improves depends on the specific task.
- Jack's Car Rental can be analyzed by comparing its ordinary state-and-action description with a description centered on the state produced by the action.