Episodic Semi-gradient Sarsa for Control

From Curves to Control

Episodic Semi-gradient Sarsa can be understood from two connected perspectives. Inside an episode, the algorithm repeatedly updates value-function weights θ after observing what happens when it takes an action. Across many episodes, its performance can be displayed as a learning curve. The first perspective explains how learning happens; the second helps us study how performance changes over time.

One Episode of Learning

Episodic Semi-gradient Sarsa processes one episode at a time. It begins with arbitrarily initialized value-function weights θ. For an episode, it obtains an initial state and action. It then takes the current action, observes a reward and a next state, and updates θ. The algorithm repeats this process until the episode ends.

The diagram shows the repeated structure of the algorithm. The current state-action pair produces an observed reward and next state. The terminal test then selects the target used to update θ. If the transition is nonterminal, the next state and next action become the current state and action for the next step.

Two Update Paths

The terminal test is not a minor bookkeeping detail. It determines whether the update uses only the observed reward or also includes an estimated value for the next state-action pair.

Tracing a Transition

Mountain Car and Tile Coding

The reported experiment combines semi-gradient Sarsa, tile-coding function approximation, and the Mountain Car example. Tile-coding function approximation is therefore the representation setting in which the value function is learned for this example. The learning process changes the value-function weights θ, and the resulting performance can be examined through learning curves.

The source identifies the components of the experiment but does not provide a detailed tile layout or a numerical example of individual tile activations. The safe conclusion is that tile coding supplies the function-approximation setting for the Mountain Car experiment; the weights θ are updated as semi-gradient Sarsa processes episodes in that setting.

Reading Figure 10.2

A learning curve represents performance over time. In this context, the curves concern semi-gradient Sarsa used with tile-coding function approximation on the Mountain Car example. Figure 10.2 displays several curves so that different step sizes can be considered together.

1. Identify the common experimental setting: semi-gradient Sarsa with tile-coding function approximation on the Mountain Car example.
2. Identify which curve is associated with which step size.
3. Compare the displayed performance over the period represented in the figure.
4. Describe only the differences that the plotted curves and their labels support.
5. Avoid asserting exact numerical values or a universal ranking when the supplied information does not provide them.

Mistakes in Control Flow

• Treating every transition as nonterminal

  The terminal test determines the update rule, and terminal transitions end the episode.

  Fix: Use the observed reward as the target component, update θ, and stop the episode.

• Using only the reward on a nonterminal transition

  A nonterminal transition includes the next action's estimated value.

  Fix: Choose A′ using q̂(S′, ·, θ), include its estimated value in the nonterminal target, and then advance to S′ and A′.

• Choosing A′ before checking whether S′ is terminal

  The terminal decision comes first. A′ belongs to the nonterminal path.

  Fix: Inspect S′, select the terminal or nonterminal rule, and choose A′ only on the nonterminal path.

• Claiming a precise result from Figure 10.2 without reading its labels

  The source identifies several curves and various step sizes but does not state exact numerical values or a particular ranking.

  Fix: Match each curve to its labeled step size and describe only the displayed comparison.

• Confusing the function-approximation setting with a detailed tile layout

  The source does not provide that detailed tile-level information.

  Fix: State that tile-coding function approximation is used with semi-gradient Sarsa in the Mountain Car example, without inventing unsupported layout details.

Practice the Decision

Key Takeaways

1. Episodic Semi-gradient Sarsa learns a value function by updating value-function weights θ during episodes.
2. The terminal test selects between two update paths: a terminal path based on the observed reward and a nonterminal path that also includes the estimated next state-action value.
3. A nonterminal transition chooses A′, updates θ, and continues with S′ and A′; a terminal transition updates θ and ends the episode.
4. The Mountain Car experiment combines semi-gradient Sarsa with tile-coding function approximation.
5. Figure 10.2 compares learning curves associated with different step sizes, so conclusions should remain tied to the labels and trends actually displayed.