Episodic Semi-gradient Control Methods
A learning curve is a way to represent performance over time.
The Question Behind the Curves
A learning algorithm is not judged only by one final result. Its performance can change as learning proceeds. A learning curve represents that performance over time, allowing you to examine how the plotted outcome changes across the period shown. In this topic, the curves concern semi-gradient Sarsa combined with tile-coding function approximation in the Mountain Car example.
The central reading task is not to memorize a claimed winner. It is to identify what was varied, identify what was measured, and then describe only the behavior displayed by the curves.
Following Performance Over Time
A learning curve gives a time-oriented view of algorithm performance. To read one, begin with the labels on the plotted axes and any legend. Then follow a curve across the represented period. The curve's changing position tells you how the displayed performance changes over time. This reading is different from stating an unsupported numerical result: the supplied description identifies the presence of several curves and different step sizes, but it does not provide exact numerical values or a definite ranking.
Treat the axis labels and legend as part of the evidence. First determine what the plotted quantity represents and what the horizontal progression represents. Only then describe whether the displayed performance changes, differs between curves, or remains difficult to distinguish.
The Mountain Car Setting
The reported experiment combines three elements: semi-gradient Sarsa, tile-coding function approximation, and the Mountain Car example. The Mountain Car example supplies the concrete problem in which learning performance is examined. Tile-coding function approximation is the representation method used with semi-gradient Sarsa in that setting. The source pack does not supply the exact tile layout, the numerical parameters, or the internal value assignments, so those details should not be inferred from the figure.
What Figure 10.2 Compares
Identifying the Comparison
Describe what Figure 10.2 compares without claiming an unsupported numerical result.
Identify the method: The curves concern semi-gradient Sarsa used with tile-coding function approximation.
Identify the problem: The method is examined on the Mountain Car example.
Identify the varied setting: Figure 10.2 shows several curves associated with various step sizes.
Identify the measured evidence: The curves show performance over the period represented in the figure.
Limit the conclusion: Describe the visible differences between the curves only after checking their labels and plotted behavior. Do not name a best step size unless the figure itself clearly supports that claim.
Figure 10.2 compares performance curves for semi-gradient Sarsa with tile-coding function approximation on Mountain Car under different step-size settings.
Why Step Sizes Produce Multiple Curves
A step size is one of the settings being varied in the reported comparison. Because Figure 10.2 considers various step sizes, the figure uses separate curves to let the reader compare the performance associated with those settings over the represented period. The important question is not whether multiple curves exist in isolation, but how the plotted performance differs when the step-size setting differs.
Before comparing curve behavior, match every curve to its step-size label. A statement about one curve cannot be transferred to another curve unless the labels show that they represent the same setting.
Mistakes in Figure Interpretation
Claiming a definite best step size without evidence from the plotted curves.
The supplied material states that Figure 10.2 contains curves for various step sizes, but it does not state a particular ranking.
Fix:
Describe the relative curve behavior only when the labels and plotted evidence support it.Treating the curve as a result for semi-gradient Sarsa alone.
The reported setting combines semi-gradient Sarsa with tile-coding function approximation on Mountain Car.
Fix:
Attribute the displayed result to the combined experimental setting.Inventing exact numerical values that are not supplied.
The source identifies the comparison but does not provide exact numerical values.
Fix:
Use qualitative descriptions tied to the visible axes, labels, and curve patterns.Describing tile coding in more detail than the evidence allows.
The supplied material identifies tile-coding function approximation as part of the setting but does not provide an internal tile map or weight values.
Fix:
State its role as the function-approximation method used in the Mountain Car experiment.
Evidence-Based Reading Practice
Suppose you are shown Figure 10.2 and asked to explain one visible difference between two curves. Write a four-part answer: name the method and problem setting, identify the two step-size labels, describe the displayed performance difference over the represented period, and state one conclusion you cannot make without additional evidence.
Hints
- Include semi-gradient Sarsa, tile-coding function approximation, and the Mountain Car example.
- Use the legend to identify which curve belongs to which step size.
- Describe only the behavior visible in the plotted curves.
- Do not invent exact values or a ranking that the figure does not establish.
A Careful Written Interpretation
Construct a defensible interpretation when the figure shows several step-size curves but the source description does not supply their numerical values.
State the setting: Say that the curves concern semi-gradient Sarsa with tile-coding function approximation on the Mountain Car example.
State the comparison: Say that various step sizes are represented by separate curves.
Report the observation: Refer to the curve labels and describe the differences visible over the period represented in the figure.
State the limit: Do not claim exact values or a universal best setting unless those claims are directly supported by the figure.
The figure compares how semi-gradient Sarsa with tile-coding function approximation performs on Mountain Car under various step-size settings. The defensible conclusion is whatever the labeled curves visibly show; unsupported numerical rankings should be omitted.
Key Takeaways
- A learning curve represents algorithm performance over time.
- Figure 10.2 concerns semi-gradient Sarsa with tile-coding function approximation on the Mountain Car example.
- Different step sizes are represented by different curves so their displayed performance can be compared.
- The curve labels, axis labels, and plotted patterns are the evidence for interpretation.
- Exact numerical results and a particular curve ranking should not be claimed unless the figure supports them.
Key Takeaways
- A learning curve shows how an algorithm's displayed performance changes over time.
- Figure 10.2 compares several step-size settings for semi-gradient Sarsa with tile-coding function approximation on Mountain Car.
- Tile coding is part of the reported function-approximation setting, while the supplied material does not specify its internal tile values.
- Reliable interpretation depends on matching curves to labels and describing only supported behavior.
- Do not infer exact numerical values or a definite ranking that the plotted evidence does not establish.