Reinforcement Learning Updates

Why One Successful Experience Is Not Enough

A reinforcement learning system learns from experience, but a useful system cannot be limited to repeating what it learned in one exact situation. Imagine a system that can use what it learns only in situations it has already encountered. Its usefulness would be sharply limited. Broad artificial intelligence and large engineering applications require generalization: using available experience to support decisions beyond one isolated learning event.

Generalization is the bridge from a single learning event to wider use. The system must learn from available experience in a way that supports action in situations that are not identical to the original experience. This requirement is why reinforcement learning needs an approximation capability rather than a process limited to isolated events.

From Backup to Training Example

The central design pattern is to connect a reinforcement-learning backup with supervised-learning function approximation. A backup produced during reinforcement learning can be treated as a training example. Instead of using the backup only as an internal reinforcement-learning update, the system supplies it as training data to an existing function-approximation method.

1. The reinforcement-learning system observes a state, selects an action, experiences a reward, and reaches a next state.
2. The system uses these elements to produce a backup.
3. The backup is treated as a training example rather than only as an internal reinforcement-learning update.
4. The training example is supplied to a supervised-learning function-approximation method.
5. The approximation method provides the generalization capability needed for use beyond the exact learning event.

Trial, Consequence, and Choice

The Law of Effect frames learning as trial and error. An agent encounters alternatives, experiences their consequences, and changes what it does through those experiences. For this lesson, the Law of Effect is best understood as a learning pattern rather than as a complete algorithm.

A rewarding action is not useful in isolation. The system must also learn when that action is appropriate. Reinforcement learning therefore combines two features: selection, which compares alternatives through their consequences, and association, which links alternatives to particular situations or states.

Context Turns Actions into a Policy

Consider a generated teaching scenario with two situations and several available actions. In the first situation, one action is followed by a better consequence than the alternatives. Selection explains why that action is favored. Association explains why the action should be connected to the first situation rather than treated as the best action everywhere. In the second situation, a different action may be favored. The learned policy is therefore a set of context-linked choices, not a single action chosen in isolation.

The key question is not only which action produces a good consequence. It is also where that action belongs. Selection compares alternatives. Association ties the alternatives found through selection to the situations or states in which they apply. Together, these connections form the agent's policy.

Common Misreadings

• Treating reinforcement learning as a system that only searches for the action with the largest reward.

  The action must also be connected to the particular situation or state in which it is useful.

  Fix: Separate selection, which compares alternatives through consequences, from association, which links an alternative to its context.

• Confusing a reinforcement-learning backup with the function-approximation method.

  The backup is treated as a training example, while the function-approximation method receives that example as training data.

  Fix: Describe the reinforcement-learning system as producing the backup and the supervised-learning method as using it for approximation.

• Assuming that learning from one event is automatically generalization.

  Broad AI and engineering applications require use beyond one isolated learning event.

  Fix: Ask how the backup is used with function approximation to support learning beyond the exact event.

• Reading the Law of Effect as a complete reinforcement-learning algorithm.

  The Law of Effect is being used here as a learning pattern involving alternatives, consequences, and changed behavior.

  Fix: Use it to understand the trial-and-error pattern, then distinguish that pattern from the particular reinforcement-learning method.

Apply the Update Pattern

The Complete Learning Picture

1. Reinforcement learning needs generalization so that experience can support action beyond one exact situation or isolated learning event.
2. A reinforcement-learning backup can be treated as a supervised-learning training example.
3. The reinforcement-learning setting produces experiences and backups; function approximation uses training examples to provide approximation capability.
4. The Law of Effect describes learning through trial and error as alternatives are followed by consequences and behavior changes.
5. Selection compares alternatives through their consequences, while association connects alternatives to the situations or states where they apply.
6. Finding a rewarding action is not the whole process because the action must also be connected to the context in which it is useful.