Behavioral Training
Shaping uses reinforcement to guide behavior through successive approximations.
Why Final Behaviors Are Difficult
Some behaviors are too complex to appear in their final form immediately. Shaping addresses this problem by reinforcing actions that gradually resemble the desired behavior. Rather than waiting for the complete performance, the trainer changes what counts as good enough as the learner gets closer.
Shaping is reinforcement guided through successive approximations toward a target behavior.
The Shaping Mechanism
The central mechanism is a changing reinforcement criterion. Early in training, an action may be reinforced because it is a rough approximation of the desired behavior. Once the learner performs that approximation more reliably, the trainer changes the standard. Reinforcement now depends on a behavior that is closer to the target. This process continues until the target behavior itself is achieved.
Shaping is therefore more than rewarding the same action repeatedly. The trainer responds to the learner's progress by progressively changing what qualifies for reinforcement. The final behavior emerges through repeated adjustments rather than through one instruction that produces the finished behavior immediately.
The Pigeon and the Wooden Ball
A source example involves training a pigeon to bowl by swiping a wooden ball with its beak. The important lesson is not that bowling is a natural or typical behavior for a pigeon. The lesson is that an unusual behavior can be developed gradually by reinforcing successive approximations instead of waiting for the complete action to occur at the beginning.
From Approximation to Bowling
How can shaping make an unusual behavior easier to develop?
Begin with an approximation: The trainer reinforces behavior that moves toward the desired interaction with the wooden ball, rather than requiring the complete bowling behavior immediately.
Reinforce a closer form: As the learner's behavior becomes closer to swiping the ball with its beak, the trainer changes the criterion so that closer behavior is now required.
Require the target form: The standard continues to become more specific until the target behavior of swiping the wooden ball with the beak is reached.
A behavior that would be difficult to expect in its final form can emerge through reinforced successive approximations.
The pigeon example demonstrates a training strategy, not a claim that the final behavior is natural for the learner.
Building Complexity Step by Step
Intermediate steps reduce the difficulty of learning a complex behavior all at once. Each approximation gives the trainer a behavior that can be reinforced and then used as the basis for a closer approximation. The learner does not need to produce the finished behavior before any progress can be recognized.
Generated illustration: imagine a learner whose desired behavior is an unusual multi-part action. Instead of waiting for every part to appear together, training can recognize an easier action that points toward the target, then recognize a closer version, and finally require the complete behavior. The details of this illustration are generated; its training logic follows the source description of successive approximations.
Shaping in Reinforcement Learning
The same general strategy can be used in computational reinforcement learning. A computational agent may receive no non-zero reward when the rewarding situation is rare or when its initial behavior cannot reach that situation. Learning directly from the final reward can then be difficult because the agent has little or no useful reward signal to guide progress.
Shaping offers another route: begin with an easier problem and increase its difficulty as the agent learns. The intermediate stages provide rewarding situations that are easier to obtain than the final one. As the agent improves, the training criterion can move closer to the original target.
Common Misunderstandings
Treating shaping as repeated reinforcement of exactly the same action.
Shaping requires the standard for reinforcement to change as the learner's behavior becomes closer to the target.
Fix:
Progressively reinforce closer approximations rather than holding the criterion fixed.Waiting for the complete target behavior before providing any reinforcement.
The source describes shaping as a response to behaviors that are too complex to appear in final form immediately.
Fix:
Recognize and reinforce behaviors that gradually resemble the target.Assuming the source example depends on the behavior being natural for the learner.
The lesson of the example is that an unusual behavior can be developed gradually.
Fix:
Focus on the changing reinforcement criteria and the successive approximations.Assuming a computational agent can always learn from the final reward alone.
When the final reward is difficult to obtain, direct learning from it can be difficult.
Fix:
Use an easier problem first and increase difficulty as the agent learns.
Apply the Idea
A desired behavior is too complex to appear immediately, and the final rewarding situation is difficult for a learner or computational agent to reach. Explain how shaping could make progress possible. Identify what would be reinforced first, how the criterion would change, and why the final reward alone might be insufficient.
Hints
- Start with an easier action or problem that resembles the target.
- Explain that the reinforcement criterion becomes stricter as behavior improves.
- Connect the intermediate rewards to the difficulty of obtaining the final reward.
What do you think happens?
A learner has mastered an early approximation. What should happen next in a shaping procedure?
Reveal answer
Answer: Raise the criterion so that a closer approximation is required.
Shaping progressively changes what counts as good enough as the learner's behavior becomes closer to the target.
Key Takeaways
- Shaping reinforces successive approximations toward a desired behavior.
- Intermediate approximations make complex or unusual behavior easier to develop than waiting for the final form immediately.
- The criterion for reinforcement changes as the learner's behavior becomes closer to the target.
- In computational reinforcement learning, shaping can help when the final rewarding situation is sparse or inaccessible.
- The pigeon-and-wooden-ball example illustrates that shaping can develop an unusual behavior through gradual reinforcement.
Key Takeaways
- Shaping guides behavior through reinforced successive approximations.
- The trainer progressively changes the reinforcement criterion as performance improves.
- Intermediate steps are useful when the final behavior is complex or unusual.
- Computational reinforcement learning can use easier intermediate problems when final rewards are sparse or difficult to reach.