Optimal Value Functions

The Ideal and the Practical Agent

In reinforcement learning, it is natural to aim for the policy that achieves the highest value. That policy is called optimal. The important qualification is that optimality describes the strongest result, not the amount of computation required to discover it. An agent may have a clear description of its environment and still be unable to calculate an optimal action before the next decision is required.

Why Knowing the Environment Is Not Enough

An environment model provides information about how the environment behaves. Computation must then use that information to evaluate possible choices, compare their values, and select an action. The difficult part may be the amount of computation needed to generate the final policy, even when the environment dynamics are completely and accurately known.

A single time step creates a practical boundary. The agent observes a state, evaluates possibilities, and must select an action. Computation that is not completed before the decision is required cannot improve that immediate decision. Therefore, the agent is constrained not only by what it knows, but also by what it can compute quickly enough.

Chess illustrates the distinction. Even large, specialized computers may not compute optimal moves. This does not mean that chess has no useful policies or that strong play is impossible. It means that substantial computational resources do not guarantee that the mathematically optimal policy can be computed.

Reading an Optimal Value

The value belongs to the state, not to one named first action. To interpret v∗(s), imagine starting at s and allowing the best available decisions to be made from there. The result is the highest value that can be achieved from that state.

State Values and Action Values

An optimal value function evaluates a state before fixing a particular first action. An optimal action-value function evaluates a state after a particular first action has been chosen. The difference is the object being evaluated: v∗(s) concerns the state, while q∗(s, a) concerns the state-action pair.

Bellman Optimality Relationships

Bellman optimality equations characterize optimal values by expressing the value of each state in terms of the choices available from that state. They create state-by-state relationships: the value assigned to a state must be consistent with the best choices available there and with the values of the states that can follow.

The recycling robot example makes the structure small enough to see clearly. Its state space contains two states, high and low, so the Bellman optimality description contains one equation for the high state and one equation for the low state. Each equation checks whether the value assigned to its state agrees with the choices available there.

It is useful to read these equations as mutual checks. The value of the high state must be consistent with the choices available in high, and the value of the low state must be consistent with the choices available in low. Solving the relationships gives the optimal values for the states.

From Values to an Optimal Policy

Once the optimal value function has been obtained, an optimal policy selects actions that achieve the maximum value from each state. The value function therefore supports policy construction: evaluate the available actions, identify those associated with the highest value, and choose one of them.

Mistakes in Interpreting Optimality

• Assuming that a perfect environment model makes the optimal policy easy to compute.

  Knowledge supplies information, but the final policy may still require extreme computation.

  Fix: Treat model accuracy and computational feasibility as separate questions.

• Treating optimality as a policy that every agent can calculate during ordinary operation.

  Optimality identifies the highest-value policy; it does not guarantee that the policy can be produced in practice.

  Fix: Use optimality as a theoretical ideal and a benchmark for judging methods.

• Confusing the value of a state with the value of a state-action pair.

  The action-value function includes a commitment to the particular first action.

  Fix: Use v∗(s) for the best value from a state and q∗(s, a) for the value after selecting a first action.

• Assuming an optimal policy must contain exactly one action in each state.

  More than one action may achieve the maximum value.

  Fix: Allow every action tied for the maximum to be part of the optimal policy.

Practice: Separate the Questions

1. Optimal value functions describe the best achievable outcome from states. Bellman optimality equations characterize those values through the available choices and resulting state values. An optimal policy selects actions that achieve the maximum value, with ties allowed. An optimal action-value function differs because it evaluates a state after a particular first action has been fixed. Finally, the theoretical optimum can remain difficult or impossible to compute within the time available for a real decision.