Policies in Markov Decision Processes
A state-value function evaluates a state under a policy.
One State, Two Evaluations
In a Markov decision process, a value function describes how valuable a situation or decision is under a policy. The key question is whether we are evaluating only the current state or evaluating one particular action chosen in that state. That single difference changes what the value function is describing.
State Value Under a Policy
A state-value function evaluates a state under a policy. In the expression for state value, the focus is the state s and the policy π. The function asks how valuable that state is when considered under that policy. It does not single out one particular initial action in the description.
Reading a state-value description
A learner says, “The value of state s under policy π evaluates one particular action taken in s.” Is this an accurate description of state value?
Identify the object being evaluated: The object is state s, not a named action.
Identify the policy: The state is evaluated under policy π.
Check for an initial action: No specific initial action is included in the state-value description.
The description is not accurate. State value evaluates state s under policy π; evaluating a specific initial action requires the action-value function.
Action Value After the First Move
The action-value function qπ(s, a) evaluates the expected return from starting at state s, taking action a, and then following policy π. Its description fixes the starting state, fixes the action taken first, and specifies that the policy is followed afterward.
The initial action and the later policy have different roles. Action a is selected explicitly at the beginning of the description. After that initial action, policy π governs what is followed afterward. Therefore, qπ(s, a) is more specific than a state-value description: it evaluates one particular action in one particular state while keeping the later policy fixed.
Reading qπ(s, a)
Interpret the statement: qπ(s, a) evaluates the expected return from starting at s, taking a, and thereafter following π.
Fix the starting state: The evaluation begins at state s.
Fix the first action: The action taken first is specifically a.
Apply the policy afterward: After the selected first action, the policy π is followed.
Identify the evaluated result: The function evaluates the expected return for that complete description.
qπ(s, a) evaluates one state-action starting situation under a specified policy for what follows.
State Versus Action Evaluation
| Question | State-value evaluation | Action-value evaluation |
|---|---|---|
| What is evaluated? | A state | A particular action taken in a state |
| Starting point | State s | State s |
| Is an initial action named? | No specific initial action is included | Action a is named and taken first |
| What policy is involved? | State s is evaluated under policy π | After action a, policy π is followed |
| Main distinction | Focuses on the state under the policy | Focuses on one action in the state, then the policy afterward |
When answering a question about value functions, state the object being evaluated before discussing the policy. For state value, say that state s is evaluated under policy π. For action value, add the missing specificity: start at s, take a, and thereafter follow π.
Mistakes in Reading qπ(s, a)
Treating qπ(s, a) as if it evaluated only state s.
The expression also fixes action a as the action taken first.
Fix:
Describe qπ(s, a) as the expected return from starting at s, taking a, and then following π.Forgetting the policy after the first action.
The action-value description includes what happens afterward: policy π is followed.
Fix:
Mention both the initial action and the policy followed afterward.Assuming the initial action is selected by the policy in the same way as every later action.
The action a is explicitly fixed in the action-value description.
Fix:
Keep action a fixed as the first action, then identify π as the policy followed afterward.Leaving out the starting state.
The description begins from a particular state s.
Fix:
Name all three parts: starting state s, initial action a, and later policy π.
Check Your Interpretation
Explain the difference between these two descriptions: evaluating state s under policy π, and evaluating qπ(s, a). Your answer should identify what is evaluated, what is fixed at the start, and what happens after the first action.
Hints
- State value does not name one specific initial action.
- Action value names action a as the first action.
- The policy π is followed after that initial action.
A complete answer
Give a complete explanation of qπ(s, a).
Name the start: Begin from state s.
Name the selected action: Take the specific action a first.
Name what follows: After that action, follow policy π.
Name the evaluation: The function evaluates the expected return for this starting state, first action, and later policy.
qπ(s, a) evaluates the expected return from starting at s, taking a, and thereafter following π.
Key Takeaways
- A state-value function evaluates a state under a policy.
- The action-value function qπ(s, a) evaluates the expected return from starting at s, taking a, and then following π.
- State-value evaluation focuses on a state; action-value evaluation focuses on a specific initial action in that state.
- A complete action-value description fixes the starting state, the first action, and the policy followed afterward.
- The essential distinction is whether a specific initial action is included.
Key Takeaways
- State value evaluates state s under policy π.
- qπ(s, a) evaluates the expected return from starting at s, taking action a, and thereafter following π.
- Action value is more specific because it names the initial action.
- To interpret qπ(s, a), track the starting state, the selected first action, and the later policy.