The Reinforcement Learning Agent

From Situation to Behavior

An agent cannot behave until its perception of the environment is connected to an action. A policy provides that connection. It describes what the agent does when it encounters a perceived state, so studying a policy means studying the rule that turns a situation into behavior.

How a Policy Selects an Action

The policy is the rule that answers this question: given the situation the agent perceives, what action should it take? The perceived situation is the input to the policy, and the selected action is its behavioral result. A policy may be represented as a simple function or lookup table, or producing the action may involve extensive computation, such as a search process. These forms differ in how the mapping is carried out, but the policy's behavioral role remains the same.

Consistent and Randomized Choices

The Agent–Environment Framework

The reinforcement learning framework describes interaction between a learning agent and an environment. The agent and environment are the two participants. States, actions, and rewards are the vocabulary used to describe their interaction: the state describes the situation being considered, the action describes what the agent does, and the reward represents the evaluative signal associated with the interaction.

• State: the situation the agent is considering.
• Action: what the agent does.
• Reward: the evaluative signal associated with the interaction.

What the Framework Preserves

The framework is more informative than a description that lists only a final answer. It keeps the decision context, the agent's behavior, and the goal-related signal together. This allows the framework to represent cause and effect, because an action is considered in relation to what happens through the interaction. It also allows for uncertainty and nondeterminism, so a decision does not have to be treated as a perfectly predictable process. Finally, rewards provide explicit goals through an evaluative signal.

A Navigation Situation

This example is not a special policy format. Its labels are invented to make the mapping visible. The important structure is that a perceived state is connected to an action, while the reward supplies an evaluative signal associated with the interaction.

Mistakes in Reading the Framework

• Treating the policy as only the action

  The action is what the agent does. The policy is the connection that maps a perceived state to an action.

  Fix: Describe the policy as the rule or mapping that connects "target ahead" with "move toward target."

• Leaving out the perceived state

  An agent's behavior requires a connection from perception of the environment to an action.

  Fix: State the situation first, then identify the action connected to it by the policy.

• Confusing the reward with the action

  The action describes what the agent does. The reward represents the evaluative signal associated with the interaction.

  Fix: Separate what the agent does from the signal that describes the desirability of the outcome.

• Assuming every policy must be a simple lookup table

  A policy can be a simple function or lookup table, but producing the action can also involve extensive computation, such as a search process.

  Fix: Focus on the policy's role as a state-to-action mapping, not on one required representation.

• Describing only a final answer

  The framework keeps the decision context, behavior, and goal-related signal together.

  Fix: Describe the situation with states, actions, and rewards and relate the action to what happens through the interaction.

Practice the Three-Part Description

The Framework in One Pass

1. Start with the state the agent perceives. Apply the policy, which connects that state to an action. The agent and environment then continue their interaction, producing a reward as an evaluative signal. This compact framework keeps behavior, cause and effect, uncertainty and nondeterminism, and explicit goals in one description.