Action-Value Functions

From Situation to Decision

In reinforcement learning, it is useful to evaluate both situations and decisions. A state tells us where the agent is, while an action tells us what the agent does from that state. Value functions describe the expected return associated with these possibilities.

The central distinction is whether the evaluation names only a state or also names a particular action. The optimal state-value function v∗(s) evaluates a state at the state level. The optimal action-value function q∗(s, a) evaluates a particular state-action pair.

Tracing an Initial Action

To interpret q∗(s, a), trace the situation in order. Begin in state s. Choose the particular action a. After that initial action, follow an optimal policy. The function evaluates the expected return for this complete description.

State-Level Optimal Value

The word maximum is important. v∗(s) asks how much return can be achieved from the state when the agent follows an optimal policy. It does not identify one particular first action in its notation. Instead, it gives a state-level view of the best expected return available from that state.

A useful way to read v∗(s) is: how much return can be achieved from this state when the agent follows an optimal policy? This differs from asking what happens after committing to one named action. The latter is the more specific question answered by q∗(s, a).

Comparing State and Action Values

v∗(s) and q∗(s, a) are related but answer different questions. v∗ describes the best expected return at the level of a state. q∗ describes the expected return after the agent commits to a particular initial action from that state and then behaves optimally.

The policy-specific versions use the same distinction. A state-value function evaluates a state under a policy. The action-value function qπ(s, a) evaluates the expected return from starting at s, taking a, and thereafter following policy π.

Reading qπ(s, a) Precisely

A complete action-value description contains three fixed components. First, identify the starting state. Second, identify the action taken first. Third, identify the policy followed afterward. Omitting any of these can change what is being evaluated.

Policy-Specific State Value

The policy matters because the value of a state is being described under a specified way of behaving. The same state-level question is different from the action-level question in qπ(s, a), where the first action is explicitly fixed before the policy is followed.

Common Interpretation Errors

• Treating v∗(s) as the value of one specific action.

  v∗(s) is indexed by a state, while q∗(s, a) is indexed by both a state and an action.

  Fix: Use q∗(s, a) when the initial action is part of the situation being evaluated.

• Describing q∗(s, a) without mentioning the initial action.

  The selected action is essential to the action-value description.

  Fix: Say that the agent starts at s, takes a, and then follows an optimal policy.

• Forgetting which policy is followed after the initial action.

  The policy followed afterward is one of the three essential parts of qπ(s, a).

  Fix: Name the subsequent policy explicitly.

• Assuming that qπ(s, a) and vπ(s) describe exactly the same information.

  State value evaluates a state under a policy, whereas action value evaluates a particular initial action in that state followed by the policy.

  Fix: Check whether the specific first action is included.

Practice the Distinction

Key Takeaways

1. v∗(s) gives the maximum expected return achievable from state s.
2. q∗(s, a) evaluates a specific initial action from state s, followed by an optimal policy.
3. v∗ provides a state-level view, while q∗ provides an action-specific view.
4. vπ(s) evaluates a state under policy π.
5. qπ(s, a) fixes the starting state, the selected initial action, and the policy followed afterward.