Concepts / Returns and State Values

Returns and State Values

Episode data records what happened after states were observed.

  • Programming

From Episodes to Predictions

A completed episode tells us what happened after a state was observed. That later outcome is the return associated with the observation. By collecting the returns associated with a state, we can form an estimate of that state's value. In this exercise, the goal is to turn the episode record into predictions for V(A) and V(B).

appears inprovidescontributes toObserved stateCompleted episodewhat happened afterwardReturn0 or 1State estimateV(state)
After a state is observed in an episode, what outcome or return is associated with it, and how does that data move into the estimate?

Tracing the Recorded Episodes

There are eight episodes altogether. Only the first begins in A. That first episode moves from A to B with reward 0 and then terminates from B with reward 0. The other seven episodes begin in B and end immediately.

moves tolater returnthen terminatesends immediately withAfirst episodeBsame episode0return from A0return from B in firstepisodeBseven other episodes1 or 0immediate episode outcomes
What happens after each state is visited, and how are the state and its later outcome connected?

A state contributes the outcome that follows its observation, not merely the state name itself. The first episode therefore supplies a return of 0 for A and also a return of 0 for its later observation of B.

Counting Returns by State

Estimating V(A) and V(B)

Use the eight recorded episodes to determine the supported estimates for V(A) and V(B).

Collect A's observations: Only the first episode begins in A. Its return is 0, so A has one observed return and that return is 0.

Estimate A's value: Because the only observed episode beginning in A has return 0, the batch supports V(A) = 0.

Collect B's observations: B is observed in the first episode after the transition from A and also at the beginning of the other seven episodes. This gives eight observations of B altogether.

Count B's returns: Across those eight observations, B has six returns of 1 and two returns of 0.

Estimate B's value: The supported estimate for B is 3/4 because three quarters of B's eight observed returns are successful: six of the eight returns are 1.

V(A) = 0 and V(B) = 3/4 for this batch of episodes.

providessupportsprovidessupportsAone observation0one returnV(A) = 0Beight observations1, 1, 1, 1, 1, 1, 0,0six 1s and two 0sV(B) = 3/4
Which episode outcomes belong to state A or state B, and how are they combined to produce each state estimate?

Why B Equals Three Quarters

The value 3/4 comes directly from the eight recorded observations of B. Six observations are followed by a return of 1, while two are followed by a return of 0. Thus, the successful outcomes account for six of the eight observations, which is 3/4.

countscountscountscountscountscountsdoes not add a successful returndoes not add a successful returnB1return 13/4six successful returns outof eightB2return 1B3return 1B4return 1B5return 1B6return 1B7return 0B8return 0
Which four observations of state B are being counted, and which three contribute a successful outcome to make the estimate 3/4?

Do not count only the seven episodes that begin in B. The first episode also contains an observation of B after the move from A. Including that observation gives eight B observations in total.

Evidence Versus Guesswork

An observed state estimate must be tied to the returns recorded after that state. For A, the batch contains one observation and its return is 0, so V(A) = 0 is supported. For B, the batch contains eight observations with six returns of 1 and two returns of 0, so V(B) = 3/4 is supported. A different value with no corresponding observations in the episode record would be an unsupported guess.

hassupportscannot supportObserved stateepisode record existsRecorded returnsA: 0; B: six 1s and two 0sSupported estimateV(A) = 0 or V(B) = 3/4Unobserved evidenceno corresponding returnsUnsupported guessno episode support
What evidence from the recorded episodes supports an estimate for a state, and what would be an unsupported value with no corresponding observations?

  • Counting only episodes that begin in B.

    The first episode begins in A but later observes B before terminating. That observation also contributes a return for B.

    Fix: Count every observation of B, including the B observation inside the first episode. There are eight B observations.

  • Using the return for A as the return for every state in the episode.

    Each observed state is connected to what happens after that observation. The first episode supplies a return of 0 for A and a return of 0 for B.

    Fix: Track the state observations separately and associate each with its later outcome.

  • Choosing a value without evidence in the batch.

    The exercise asks for estimates supported by completed episodes, not unsupported guesses.

    Fix: Base each estimate on the returns recorded after observations of that state.

Check Your Estimate

MEDIUM

Explain in your own words why the batch supports V(A) = 0 and V(B) = 3/4. In your explanation, state how many observations belong to A, how many belong to B, and how many of B's observations have return 1.

Hints
  • Start with the only episode that begins in A.
  • Remember that the first episode later observes B.
  • For B, distinguish the six returns of 1 from the two returns of 0.

What do you think happens?

Before checking the episode record, how many observations of B should be included in the estimate for V(B)?

  • One
  • Seven
  • Eight
  • Only the observations with return 1
Reveal answer

Answer: Eight

Seven episodes begin in B, and the first episode also observes B after moving from A. The first episode's B observation must be included.

Key Takeaways

  1. Episode data records what happened after states were observed.
  2. Only the first episode begins in A, and its return is 0, so the batch supports V(A) = 0.
  3. B is observed eight times: once in the first episode and once at the beginning of each of the other seven episodes.
  4. B has six returns of 1 and two returns of 0, giving the supported estimate V(B) = 3/4.
  5. An estimate is supported when it is based on recorded returns; a value with no corresponding observations is an unsupported guess.

Key Takeaways

  • A return is the outcome associated with what happens after a state is observed in a completed episode.
  • The only observed episode beginning in A has return 0, so V(A) = 0 for this batch.
  • B has eight observations, with six returns of 1 and two returns of 0.
  • Because six of eight B observations have return 1, the supported estimate is V(B) = 3/4.
  • State estimates must be grounded in recorded episode evidence rather than unsupported guesses.

Related Topics