Evolutionary Methods

Two Ways to Search for Better Behavior

Reinforcement learning methods can search for better behavior from different starting points. A policy gradient method searches through policies defined by numerical parameters. It uses interaction with the environment to estimate a direction for changing those parameters. An evolutionary method takes another route: it compares the complete behavior of several candidate policies and favors candidates that obtain more reward over their interaction period.

Searching Through Policy Parameters

A policy gradient method searches over parameter-defined policies rather than treating a value estimate as the primary search object. The policy is described by a collection of numerical parameters. Those parameters determine the policy the agent uses, so changing the parameters changes the policy itself.

The diagram separates three ideas that can otherwise be confused. Policy gradient search changes the numerical parameters that define a policy. Value-function estimation focuses on how valuable a state or action is. Evolutionary search compares candidate policies by the behavior those policies produce over an interaction period. These ideas can appear together in reinforcement learning, but they are not identical.

From Interaction to Parameter Adjustment

Environment interaction matters because the method needs evidence about how the current policy behaves. The policy uses its current numerical parameters, the agent interacts with the environment, and the resulting behavioral interactions provide information for estimating a direction in which the parameters should move. The method then adjusts the parameters and searches again with another parameter-defined policy.

The Role of Value Estimates

Value functions are not required in every method in this family. Some policy gradient methods use value-function estimates to improve their gradient estimates. In that situation, the value estimate is not the same thing as the policy-gradient estimate. It helps make the estimated direction for changing policy parameters more useful.

Evaluating a Population of Policies

Evolutionary methods approach reinforcement learning by comparing complete behaviors rather than constructing a value estimate. Imagine a collection of agents that use different policies. Each agent interacts with the environment during its lifetime. After that interaction period, the agent is evaluated by the reward obtained from its complete behavior. Candidates with more reward are favored as the basis for further search, while weaker candidates are not favored for the next stage.

Lifetime Performance Versus Value

An evolutionary method can compare whole policies even when it never constructs a value function. Its important unit of evaluation is the agent's lifetime behavior. A value function instead estimates the value of a state or action. These approaches are different, although both can be used in reinforcement learning and some policy gradient methods can use value estimates to improve their gradient estimates.

When Evolutionary Search Helps

Evolutionary methods may be effective when the policy space is small, when the policy space can be organized so that good policies are common or easy to find, or when substantial time is available for searching. These conditions make it more plausible that repeated comparison and selection will encounter useful policies.

Common Reasoning Errors

• Saying that a policy gradient method primarily searches over value estimates.

  The primary search object is a policy defined by numerical parameters.

  Fix: State that the method estimates a direction for adjusting the parameters that define the policy.

• Claiming that every policy gradient method must use a value function.

  Value estimates can improve gradient estimates in some methods, but they are not required in all of them.

  Fix: Describe value estimates as optional aids in some policy gradient methods.

• Treating an evolutionary method's score as a value function.

  The evolutionary method evaluates complete lifetime behavior, whereas a value function estimates the value of a state or action.

  Fix: Call the lifetime result a reward-based comparison of candidate policies.

• Assuming each agent must learn during its lifetime for the method to be evolutionary.

  The source describes non-learning agents whose complete behaviors are evaluated, followed by selection in the broader search.

  Fix: Separate individual lifetime behavior from the broader process that favors stronger candidates.

Practice the Distinction

Key Takeaways

1. Policy gradient methods search through policies defined by numerical parameters.
2. Environment interaction provides evidence for estimating a direction in which those parameters should be adjusted.
3. Value-function estimates can improve gradient estimates in some policy gradient methods, but they are not required in every method.
4. Evolutionary methods compare complete policy behaviors over an interaction period and favor candidates that obtain more reward.
5. Evolutionary evaluation differs from estimating the value of a particular state or action and may be effective in small or searchable policy spaces, with substantial search time, or when accurate environmental sensing is difficult.