Policy-Gradient Methods
Function approximation introduces issues including nonstationarity, bootstrapping, and delayed targets.
Why Function Approximation Changes the Problem
Policy-gradient methods appear near the end of a chapter sequence about reinforcement learning with function approximation. The sequence does not treat function approximation as merely a new way to represent an ordinary reinforcement-learning problem. It emphasizes that function approximation introduces issues that are not normally encountered in conventional supervised learning.
The Chapter Roadmap
The early sequence first restricts attention to on-policy training. Chapter 9 addresses prediction: the policy is given, and the value function is the part being approximated. Chapter 10 moves to control, where the aim is to find an approximation to the optimal policy. Chapter 11 broadens the setting to off-policy methods. Chapter 12 changes emphasis to eligibility traces and their computational role in multistep methods. The final chapter presents policy-gradient methods as a different, direct approach to approximating the optimal policy.
| Stage | Primary focus | Question it addresses |
|---|---|---|
| On-policy prediction | Given policy and approximated value function | How can the value function for the given policy be approximated? |
| On-policy control | Approximation to the optimal policy | How can learning move toward an optimal policy? |
| Off-policy methods | Broader training setting | How does learning change when the training setting is broadened beyond on-policy training? |
| Eligibility traces | Computational mechanism for multistep methods | How can eligibility traces improve the computational properties of multistep methods? |
| Policy-gradient methods | Direct policy approximation | How can the optimal policy be approximated directly? |
The roadmap separates training settings from algorithmic mechanisms and control approaches.
Three Function-Approximation Warnings
The overview names three issues that arise when reinforcement learning uses function approximation: nonstationarity, bootstrapping, and delayed targets. At this stage, the important lesson is not a detailed algorithm for each term. Instead, the terms signal that the learning problem may behave differently from conventional supervised learning and must be examined again.
The three terms are best used as diagnostic questions: could the learning situation change while learning proceeds, could current estimates affect later learning targets, and could useful targets arrive after a delay? The source overview identifies these as issues, but does not require memorizing a detailed algorithm for each one at this point.
The Policy-Gradient Interaction Loop
A policy-gradient method searches over policies defined by numerical parameters rather than treating a value estimate as the primary search object. The parameters define the policy currently being used. The agent then interacts with the environment, and the resulting experience provides information for estimating a direction in which the parameters should move. After adjustment, the parameters define another policy, and the interaction-and-adjustment process can be repeated.
Tracing One Parameter Adjustment
Suppose a policy is represented by numerical parameters and is used to interact with an environment. Trace what a policy-gradient method does after that interaction.
Start with a policy: The current numerical parameters define the policy that the agent uses.
Gather experience: The agent interacts with the environment. The resulting behavioral interactions provide details about the policy's performance.
Estimate a direction: The method uses the experience details to estimate a direction for changing the policy parameters so that performance may improve.
Adjust the parameters: The parameters are moved in the estimated direction. This produces another parameter-defined policy.
Search again: The updated policy can interact with the environment again, continuing the search through parameter-defined policies.
The central state change is from one parameter-defined policy to another. Environment interaction supplies the information used to estimate the adjustment direction.
From Parameters to Policy Search
The object being searched is a policy described by numerical parameters. The important change is therefore not a switch between two named algorithms. It is a change in the numerical parameters that define the policy. Before an adjustment, the agent follows one parameter-defined policy; after an estimated improvement direction is applied, the parameters define another policy.
Policy-gradient methods search directly through possible policies. Their central estimate is a direction for changing the policy's numerical parameters, with the goal of improving policy performance.
The Role of Value Estimates
Policy-gradient methods do not all require a value function. The source describes them as a direct approach that may never form an approximate value function. However, some methods use value function estimates to improve their gradient estimates. In those methods, the value estimate supports the estimated improvement direction; it does not replace the policy-gradient objective.
| Question | Policy-gradient role | Value-estimate role |
|---|---|---|
| What is being searched? | Policies defined by numerical parameters | Not the primary search object in the direct policy-gradient description |
| What does the environment provide? | Experience used to estimate a parameter-adjustment direction | Information that can support a more useful gradient estimate |
| Is it required? | The policy-gradient approach is the defining method | No; value function estimates are not required by every method |
| What is the result? | A direction for changing policy parameters | An improvement to the estimated direction in some methods |
Policy Gradients and Evolutionary Search
The source-grounded distinction is that policy-gradient methods estimate a direction for changing policy parameters from the agent's interaction with the environment. The source pack does not specify an evolutionary algorithm or provide a detailed account of evolutionary parameter search, so no stronger procedural contrast should be inferred here.
When comparing the families, state only the distinction supported by the chapter overview: policy-gradient methods are characterized here by environment-derived gradient information used to adjust policy parameters. Do not claim that every parameter-search method with a population, variation, or evaluation step is completely unrelated; the boundary is not absolute in the learning objective.
Common Misreadings
Treating policy-gradient methods as ordinary value-function search.
The chapter overview defines policy-gradient methods as searching over parameter-defined policies and directly approximating the optimal policy.
Fix:
Describe the numerical policy parameters as the primary object being adjusted.Ignoring environment interaction.
The estimated direction for changing the policy parameters comes from interaction with the environment.
Fix:
Trace the flow from policy parameters to policy, interaction, experience, estimated direction, and updated parameters.Assuming every policy-gradient method must form a value function.
The overview says that policy-gradient methods need never form an approximate value function, although some may be more efficient if they approximate one as well.
Fix:
Treat value estimates as optional support that can improve gradient estimates in some methods.Confusing the chapter roadmap's purposes.
The roadmap presents eligibility traces as an algorithmic mechanism with a computational role in multistep methods.
Fix:
Separate training settings from mechanisms and control approaches.
Check Your Understanding
A learner says: Policy-gradient methods search for the best value estimate, then derive a policy from it. Rewrite the statement so that it matches the chapter overview. Include what the method searches over, where its adjustment direction comes from, and whether a value function is always required.
Hints
- Start with the phrase policies defined by numerical parameters.
- Mention interaction with the environment as the source of information for the estimated direction.
- Use the distinction between required and potentially useful value estimates.
Place these topics in the chapter sequence and state the learning objective of each: prediction, control, off-policy methods, eligibility traces, and policy-gradient methods.
Hints
- The first two topics belong to the early on-policy sequence.
- Off-policy methods broaden the training setting.
- Eligibility traces emphasize a computational mechanism.
- Policy-gradient methods provide a direct approach to approximating the optimal policy.
Key Takeaways
- Policy-gradient methods search through policies defined by numerical parameters rather than treating a value estimate as the primary search object. Experience from interaction with the environment is used to estimate a direction for changing those parameters. The parameter adjustment defines another policy, so learning proceeds as a repeated search through policies. Value function estimates are optional: some methods do not form one, while others use one to improve gradient estimates. In the broader chapter roadmap, prediction comes before control, off-policy methods broaden the setting, eligibility traces provide a multistep mechanism, and policy-gradient methods offer a direct control approach. Function approximation supplies the wider context and brings the warning signs of nonstationarity, bootstrapping, and delayed targets.
Key Takeaways
- Function approximation introduces nonstationarity, bootstrapping, and delayed targets as issues that require renewed analysis.
- The chapter sequence moves from on-policy prediction and control to off-policy methods, eligibility traces, and direct policy-gradient control.
- A policy-gradient method searches through policies defined by numerical parameters.
- Environment interaction supplies the experience used to estimate how policy parameters should be adjusted.
- Value function estimates can improve gradient estimates in some methods, but they are not required by every policy-gradient method.