Policy Parameterization for Continuous Actions

From State to Action Distribution

A policy for a continuous scalar action does not assign a separate probability to every possible real number. Instead, it defines a normal probability density over the possible action values. The density is parameterized by a mean and a standard deviation, and those parameters can be produced by parametric function approximators.

Tracing a Continuous Policy

Suppose the action is a real-valued scalar, such as a single control setting. The policy represents possible values of that action with a normal density. The full density p(x) is the policy representation: it describes how probability is distributed across possible action values.

Density Versus Probability

The value p(x) at one action value is a density, not the probability of taking exactly that action value. A density describes how probability is distributed locally across the action axis. Probability is obtained over a range of action values: it is the area under the density across that interval.

Changing the Mean

The mean is one of the parameters that selects the normal density used by the policy. Changing the mean produces a different density and shifts where the density is centered along the action axis. In policy terms, changing the mean changes the central action region represented by the distribution.

Changing the Standard Deviation

The standard deviation is the second policy parameter that selects the normal density. Changing it produces a different density with a different spread. A larger spread distributes the density across a wider region of action values, while a smaller spread concentrates it more narrowly. Thus, the standard deviation controls how broadly the policy represents possible scalar actions.

Function Approximators as Policy Parameterizers

For a continuous-action policy, parametric function approximators provide the mean and standard deviation of the normal density. The observed state is passed through the approximator, and its outputs determine which normal density represents the policy for that state. A different state can therefore lead to different parameter values and a different normal density.

This separates two jobs. The function approximator supplies the policy parameters. The normal density uses those parameters to represent how probability is distributed over real-valued scalar actions. The policy is not merely the mean by itself; it is the density selected by both the mean and the standard deviation.

Mistakes to Avoid

• Treating p(x) as the probability of one exact action value.

  p(x) is a density at a point. Probability is associated with the area under the density across an interval.

  Fix: For a probability question, identify an action range and reason about the area under the density over that range.

• Describing the policy only by its mean.

  The normal density is selected by both its mean and its standard deviation.

  Fix: Track both outputs from the parametric function approximator.

• Confusing a parameter change with a probability query.

  Changing the mean produces a different density; an interval probability is then obtained from the area under that selected density.

  Fix: First identify the density parameters, then distinguish the resulting density from any interval-based probability question.

Practice the Trace

Key Takeaways

1. A continuous scalar-action policy can be represented by a normal probability density over real-valued actions.
2. The value p(x) at one point is a density, not the probability of one exact action value.
3. Probability over an action range comes from the area under the density across that interval.
4. The mean changes the location of the normal density, while the standard deviation changes its spread.
5. Parametric function approximators can receive a state and provide the mean and standard deviation that select the policy density.