Deterministic Policies

A Better-Policy Search

When searching for a better policy, a different action should not be judged only by how attractive it looks in isolation. The important question is what happens when that action is chosen in state s and the existing policy is followed afterward. The value function gives the baseline needed for this comparison: vπ gives the value of following policy π from state s.

Two Behaviors from State s

There are two behaviors to compare. In the first, the existing policy π is followed immediately from state s. Its value is vπ(s). In the second, a proposed action a is selected in state s, and policy π is followed thereafter. The proposed behavior is therefore not just action a by itself. It is the combined behavior of choosing a now and using the existing policy as the continuation.

The Policy Evaluation Criterion

The policy evaluation criterion asks whether the proposed behavior has greater value than the current baseline. In the usual notation, qπ(s,a) represents the value of selecting action a in state s and then following π, while vπ(s) represents the value of following π from s. A proposed change is supported when qπ(s,a) is greater than vπ(s). If the proposed behavior does not compare favorably with vπ(s), the criterion does not support the change.

From a Promising Choice to Improvement

The Policy Improvement Theorem connects the one-state comparison to a new deterministic policy. If selecting the proposed action once in state s and then following π is better than following π from the beginning, the reasoning is that selecting that action every time state s is encountered should be better still. The new policy applies the proposed choice consistently. For deterministic policies, this is identified as a policy-improvement result.

Mistakes in Policy Comparison

• Judging an action in isolation

  The evaluation concerns action a followed by policy π, not action a separated from its continuation.

  Fix: Compare the complete proposed behavior with vπ(s).

• Using the wrong baseline

  The value function vπ(s) is the baseline for the current policy.

  Fix: Start the comparison with the value of following π immediately from s.

• Treating a promising one-step change as unrelated to the new policy

  The Policy Improvement Theorem explains why applying the proposed choice consistently produces a policy-improvement result for deterministic policies.

  Fix: Connect the one-step comparison to the new policy that consistently selects the proposed action.

Apply the Criterion

In this case, the proposed behavior is equal in value to the current baseline, not greater. Therefore the strict policy evaluation criterion described here does not support the change as an improvement. The theorem's relevant condition is that the proposed behavior be at least as good when considering the new deterministic policy; the source emphasizes that a greater proposed value supports the change and that the new policy is no worse when its choices are at least as good as the current policy.

Key Takeaways

1. vπ(s) is the value of following the existing policy π from state s.
2. A candidate action must be evaluated together with the continuation of following π afterward.
3. The policy evaluation criterion compares the candidate behavior with vπ(s); a greater candidate value supports the change.
4. The value function serves as the baseline for searching for a better policy.
5. The Policy Improvement Theorem explains why a promising one-step choice can support a new deterministic policy that applies that choice consistently.