Estimated Action Values
Exploration chooses a nongreedy action to improve knowledge about its value.
The Choice Behind Every Action
When several actions are available, a learner must decide whether to choose the action that currently appears best or choose another action to learn more about it. The first choice uses current knowledge to pursue reward now. The second may give up some immediate reward so that later decisions can be better informed. This is the exploration-exploitation trade-off.
Reading Current Action Estimates
An estimated action value represents the learner's current view of how rewarding an action is expected to be. When one action has the highest current estimate, that action is called greedy. A nongreedy action is any available action that does not currently have the highest estimate.
| Choice | What it does | Time scale |
|---|---|---|
| Greedy action | Uses the action with the highest current estimate | Current step |
| Nongreedy action | Chooses an action that does not currently have the highest estimate | Can improve knowledge for later steps |
Immediate Reward Versus Future Knowledge
Choosing between a known option and an uncertain option
A learner has two available actions. Action A currently has the highest estimated value. Action B is nongreedy, but its value is uncertain and it might be better than the current estimate suggests. Which action should the learner choose?
Evaluate exploitation: Choosing Action A is exploitation. Because Action A is greedy, this choice focuses on maximizing the expected reward on the current step.
Evaluate exploration: Choosing Action B is exploration. This choice may sacrifice some immediate reward, but it can improve the learner's knowledge about Action B.
Consider future decisions: If Action B turns out to be better, improved knowledge about it can support higher reward on later steps, especially when many future decisions remain.
Make the time-scale distinction: The best choice depends on the objective being considered. Action A is appropriate when the goal is current-step expected reward. Action B can be valuable when learning may improve later decisions.
Exploitation emphasizes immediate expected reward, while exploration considers whether information gained now can improve total reward later.
How Exploration Changes Later Decisions
Exploration is a decision to choose a nongreedy action in order to improve knowledge about its value. The important change is not only the reward received on the exploratory step. The learner also gains information that can affect the estimates used on later steps. If the explored action turns out to be better, that improved knowledge can support higher reward in the future.
What do you think happens?
An action is currently nongreedy but uncertain, and many future decisions remain. What is the main possible benefit of choosing it now?
Reveal answer
Answer: It improves knowledge that may support better later decisions.
Exploration can sacrifice some immediate reward to improve knowledge about a nongreedy action. If that action turns out to be better, the improved knowledge can support higher reward on later steps.
Knowledge Before and After Exploration
Consider a generated scenario with two actions. Action A currently has the highest estimated value, so it is greedy. Action B has a lower current estimate, so it is nongreedy, but the learner is less certain about its value. If the learner chooses B to explore, the immediate result may be less rewarding than the result expected from A. However, the observation provides information about B. The learner can then make later decisions with better knowledge than before.
When analyzing an action choice, ask two separate questions: Which action has the highest current estimated value, and how valuable could it be to learn about an uncertain action before making many future decisions? The first question identifies the greedy choice. The second evaluates whether exploration may improve total reward.
Mistakes in Action Selection
Treating the action with the highest estimate as the best choice for every time scale.
The greedy action is best according to current knowledge for immediate expected reward, but exploration can improve total reward over later steps.
Fix:
Separate the goal of maximizing the current step from the goal of improving future decisions through learning.Calling every nongreedy choice irrational.
Exploration deliberately chooses a nongreedy action to improve knowledge about its value.
Fix:
Ask whether the possible value of learning justifies sacrificing some immediate reward.Assuming exploration guarantees a higher total reward.
Exploration can produce greater total reward when an uncertain action may be better, but the source describes this as a possibility rather than a guarantee.
Fix:
Describe exploration as a way to improve knowledge and create the possibility of higher later reward.Confusing immediate expected reward with total reward over later decisions.
Exploration may sacrifice immediate reward while improving knowledge that affects future choices.
Fix:
State clearly whether the analysis concerns the current step or a longer sequence of decisions.
Practice the Trade-Off
A learner has several available actions. One action currently has the highest estimated value. Another action has a lower estimate but is uncertain, and many future decisions remain. Explain the case for exploitation and the case for exploration. Then state what additional consideration determines which time scale should guide the decision.
Hints
- Identify the greedy action from the current estimates.
- Explain what immediate reward exploitation prioritizes.
- Explain how exploration could improve knowledge about the uncertain action.
- Distinguish current-step reward from possible total reward over later steps.
A concise analysis
Action A has the highest current estimate. Action B is nongreedy and uncertain, and many future decisions remain.
Identify the greedy action: Action A is greedy because it currently has the highest estimated value.
Describe exploitation: Selecting Action A exploits current knowledge and focuses on immediate expected reward.
Describe exploration: Selecting Action B explores because it chooses a nongreedy action to improve knowledge about B's value.
Compare time scales: Because many future decisions remain, information about B may support higher reward later if B turns out to be better.
Action A is the exploitation choice for current-step expected reward; Action B is the exploration choice when the value of learning may improve later decisions.
Key Takeaways
- A greedy action has the highest current estimated action value.
- Exploitation chooses a greedy action based on current knowledge and focuses on immediate expected reward.
- Exploration chooses a nongreedy action to improve knowledge about its value.
- Exploration can improve total reward when an uncertain action may be better and many future decisions remain.
- A sound analysis considers both the reward available now and the value of information for later decisions.
Key Takeaways
- Estimated action values express the learner's current expectations about available actions.
- The greedy action has the highest current estimate; choosing it is exploitation.
- A nongreedy action can be selected for exploration when learning about its uncertain value may improve future decisions.
- Exploitation and exploration optimize different time scales: immediate expected reward versus possible total reward over later steps.
- The exploration-exploitation trade-off requires weighing current reward against the value of improved knowledge.