Evaluating Learned Policies
A learned policy is only worth keeping if it produces better behavior than what came before.
Why Judge a Policy by Its Flights
A learned policy is only worth keeping if it produces better behavior than what came before. Learning is not judged by how it looks on paper but by what the system actually does. In the thermal-soaring setting described in this material, the test is concrete: watch what happens to the glider's altitude as it follows a trajectory. If the altitude goes down, the behavior is failing. If the altitude goes up, the behavior is succeeding.
This article follows one comparison from start to finish. First you will see how a glider behaves when its actions are chosen at random, then how it behaves after learning, and finally how to read the two flight paths side by side so that you can say which policy performs better.
Thermal Soaring Defined
Thermal soaring is a method of gaining altitude by exploiting rising air currents.
The key idea is that the glider does not gain altitude by its own power alone. It uses rising air currents, which push it upward when it is positioned within them. The whole task is therefore about flying in a way that stays within useful rising air rather than drifting out of it.
Random Actions and Descent
Start with the untrained case. Before learning, the agent selects its actions randomly. Each control choice is essentially a guess, with no information about which choices keep the glider in rising air.
What do you think happens?
What do you think happens to the glider's altitude when its actions are chosen at random?
Reveal answer
It descends.
The source states that before learning, random actions lead to descent in the described trajectory. Random control choices do not reliably keep the glider within a rising air current, so the flight path ends lower than it started.
The reason the outcome is descent rather than climb is that nothing in the random choices is tied to the rising air. The glider is not repeatedly placed back into the current that could lift it. Over the course of a flight, the path drifts downward, and the altitude falls with it.
Before and After Trajectories
The central comparison is between two flights of the same glider in the same thermal-soaring task, differing only in whether the policy was learned. The diagram below sets the two outcomes next to each other. The untrained path with random actions ends in descent. The learned path forms a spiral, and that spiral is what allows the glider to gain altitude.
Notice what changes and what does not. The task and the glider are the same in both rows. What changes is the policy that selects actions, and the visible result is the direction of altitude change. The before-learning row ends in descent; the after-learning row ends in gained altitude.
Reading the Spiral Path
After learning, the trajectory takes the shape of a spiral, and the source identifies this spiral as the path that allows the glider to gain altitude. Think of the spiral as the signature of successful behavior: the glider stays in a repeating pattern while its height increases over the course of the flight.
Tracing a Policy-Performance Figure
The figure can be read without inspecting every individual action. The method is to trace the result in two stages, and the diagram below shows the order of those stages.
Reading the thermal-soaring comparison
Two gliders fly the same thermal-soaring task. One chooses actions randomly, and the other follows a learned policy. Decide which policy performs better using the flight paths and altitude changes.
- 1Identify the action-selection condition
The first glider uses random actions, which is the before-learning condition. The second glider uses learned behavior, which is the after-learning condition.
- 2Inspect the resulting flight path
The random-action path has no spiral and drifts downward. The learned path is a spiral.
- 3Check the altitude change
The random-action path ends in descent. The learned path gains altitude.
- 4Draw the conclusion
The learned policy produces better behavior in this task, because its trajectory supports climbing while the random policy's trajectory does not.
The learned policy performs better. Its spiral trajectory gains altitude, whereas random actions lead to descent.
Mistakes When Judging a Policy
Common mistake Assuming a learned policy always gains altitude
Why it failsThe source says the performance result depends on the trajectory described for the learned case. Not every flight path gains altitude automatically.
Do this insteadCheck the altitude change of the specific learned trajectory before concluding that it performs well.
Common mistake Judging a policy from one action instead of the whole trajectory
Why it failsPerformance is judged by what happens to the glider's altitude as it follows a trajectory, not by any single control choice.
Do this insteadCompare the full flight path and its altitude result across the before and after cases.
Common mistake Treating a spiral as only a visual detail
Why it failsThe spiral is the specific trajectory the source associates with gaining altitude after learning, so its shape is part of the evidence.
Do this insteadRead the spiral as the behavior that supports climbing, and confirm it against the altitude change.
Common mistake Forgetting that the before and after cases share the same task, so differences in the outcome come from the policy
Practice Reading the Two Outcomes
A glider flies one thermal-soaring task twice. In the first flight it chooses every action at random. In the second flight it follows a learned policy that produces a spiral. Using only the altitude results, which flight is evidence that the learned policy performs better, and why?
Hints
- Look at the direction of altitude change in each flight, not at the number of actions.
- The before-learning case is associated with descent, and the after-learning spiral is associated with gaining altitude.
Key Takeaways
- Thermal soaring is a method of gaining altitude by exploiting rising air currents.
- Before learning, the agent selects actions randomly and the glider descends.
- After learning, the glider follows a spiral trajectory that allows it to gain altitude.
- A learned policy is judged by the trajectory it produces and the direction of the resulting altitude change, not by any single action.