Afterstate Value

From Bet to Expected Value

An action value is an estimate of how valuable an action is before its outcome is known. In the Daily-Double setting described for Watson, the action is a candidate bet. The bet can lead to different resulting states because Watson may respond correctly or incorrectly. Action value computation combines the estimated values of those possible resulting states, weighted by Watson’s in-category Daily-Double confidence.

Tracing the Two Outcomes

Action value = p_DD × v-hat(S_W + bet) + (1 − p_DD) × v-hat(S_W − bet)

The important point is that the bet amount alone does not determine the action value. The computation considers the current score, the score changes associated with a correct or incorrect response, the learned value of each resulting state, and the confidence used to weight the two possibilities.

The Learned Value Function

The value function, written v-hat, is a previously learned function that gives an estimated value for a game state. During action value computation, it is queried rather than replaced. For a candidate bet, it provides one estimate for the state associated with a correct response and another estimate for the state associated with an incorrect response. The action value then combines those estimates using p_DD and 1 − p_DD.

Action Value and Resulting-State Value

A resulting-state value is an estimate for one state after a particular outcome. In the bet computation, v-hat(S_W + bet) is the estimate for the correct-response state, while v-hat(S_W − bet) is the estimate for the incorrect-response state. An action value is different: it evaluates the candidate bet before knowing which of those outcomes will occur by combining both estimates according to p_DD and 1 − p_DD.

Assessing the Afterstate

This changes the unit being assessed. An ordinary action-value function may assess a state-action pair directly. An afterstate value function assesses the state produced by that pair. If several different state-action pairs produce the same afterstate, the function can use one shared value assessment for that resulting state.

One Afterstate, Many Routes

In reinforcement learning, different position-move pairs can sometimes lead to the same resulting position. The two pairs may begin from different positions or use different moves, yet the resulting position is identical. Because the resulting position is the same, both pairs have the same value assessment when value is assigned through the afterstate.

An ordinary action-value function may assess the two position-move pairs separately. An afterstate value function instead assesses their common resulting position directly. This avoids redundant work because the shared afterstate does not need separate value assessments merely because it was reached through different routes.

Transfer Through Shared Results

The efficiency benefit is also a learning benefit. When two different state-action pairs produce the same afterstate, learning about one pair transfers immediately to the other through that shared result. The afterstate is the link: both routes point to the same value assessment.

Mistakes in Value Assessment

• Treating the action value as the value of the bet amount alone.

  The action value also depends on the current score, the estimated values of the correct and incorrect outcome states, and the confidence used to weight those outcomes.

  Fix: Trace both resulting states and combine their estimated values using p_DD and 1 − p_DD.

• Replacing the value function with the action value calculation.

  The value function estimates the values of game states. Action value computation queries it for outcome-state estimates and then combines them.

  Fix: Keep the sequence clear: produce outcome states, query v-hat, weight the estimates, and combine them.

• Confusing one afterstate value with an action value.

  That is one resulting-state estimate. The action value accounts for both correct and incorrect response outcomes.

  Fix: Use both v-hat(S_W + bet) and v-hat(S_W − bet) with their respective weights.

• Assessing identical afterstates separately.

  An afterstate value function assesses the shared resulting position directly, so separate assessments repeat the same work.

  Fix: Use the common afterstate as the shared input to the value function.

Check Your Understanding