On-Policy Control with Sarsa

Learning Without a Simulator Model

An acrobot is a two-link robotic arm. In the swing-up task, a reinforcement learning agent applies torques until the goal is reached. The important feature is how the agent learns: it controls the simulated arm through repeated interaction without access to the simulator itself. This makes the task an online, model-free control problem, treated as though the agent were interacting with a physical acrobot.

The Four-Part Acrobot State

The controller needs a description of the acrobot's current condition. The direct representation contains four continuous quantities: the position of the first link, the velocity of the first link, the position of the second link, and the velocity of the second link. They are written as θ1, θ̇1, θ2, and θ̇2.

From Continuous State to Action Value

The controller combines tile coding, linear function approximation, Sarsa(λ), and replacing traces. Tile coding supplies a representation for the continuous four-dimensional state space. Linear function approximation then uses that representation to approximate the value function rather than storing a separate exact value for every possible point in the region.

Action Choice and Episode Reset

Every episode begins with both links hanging straight down and at rest. The learning agent then applies torques. Interaction continues until the goal is reached. At that point, the acrobot returns to the same initial resting position and another episode begins.

The action set is small and discrete, so the setup uses a separate set of tilings for each action. This means the representation is not only a shared description of the physical state. Each available action has its own tiling set associated with evaluating that action in the state.

Traces in the Learning Configuration

Replacing traces are another component of the stated Sarsa(λ) configuration. Their role should be kept distinct from the role of tile coding. Tile coding represents the continuous state, action-specific tilings distinguish the discrete actions, and replacing traces belong to the learning configuration used alongside linear function approximation and Sarsa(λ).

Check Your Understanding

• Treating the acrobot state as a small list of individually enumerated states.

  The direct state is continuous, and its possible values form a bounded region in four dimensions.

  Fix: Use tile coding to represent the continuous state space and linear function approximation to approximate values.

• Assuming one shared tiling set is the complete action representation.

  The stated setup uses a separate set of tilings for each discrete action.

  Fix: Keep the physical state description distinct from the action-specific tiling sets used for action evaluation.

• Confusing replacing traces with tile coding.

  Tile coding represents the continuous state; replacing traces are part of the Sarsa(λ) learning configuration.

  Fix: Assign representation to tile coding and learning configuration to replacing traces.

• Forgetting the reset between episodes.

  The acrobot is returned to its initial resting position after the goal is reached.

  Fix: Trace each episode as resting start, repeated torque application, goal reached, and return to the initial resting position.

The Complete Design

1. The acrobot swing-up task is online and model-free because the agent learns through interaction without access to the simulator itself.
2. The direct state contains θ1, θ̇1, θ2, and θ̇2: the positions and velocities of the two links.
3. Tile coding represents the continuous four-dimensional state space, while linear function approximation approximates the value function.
4. Each discrete action has its own tiling set, allowing the controller to distinguish action-specific value estimates.
5. Replacing traces are included in the Sarsa(λ) learning configuration, and each episode resets the acrobot after the goal is reached.