Exploration vs Exploitation
Greedy actions are those with the greatest current estimated value.
One Choice, Two Purposes
When an agent must choose an action, it has two different ways to use its current situation. It can rely on what it currently believes about the actions and seek the best expected reward for the next step. Or it can choose an action to learn more about it. These purposes create the exploration-versus-exploitation problem.
A single action selection cannot both explore and exploit. Selecting the currently highest-valued action pursues immediate reward; selecting a different action creates an opportunity to improve knowledge about that action.
Finding the Greedy Action
The agent has an estimated value for each available action. The greedy action is the action with the greatest current estimated value. “Greedy” describes the choice at the present decision, not a guarantee that the action will always produce the greatest reward.
Comparing three estimated action values
Suppose an agent currently estimates the values of three actions as A = 4, B = 7, and C = 5. Which action is greedy?
List the estimates: The current estimates are 4 for A, 7 for B, and 5 for C.
Locate the greatest estimate: The greatest current estimated value is 7.
Identify the action: Action B has the greatest estimated value, so B is the greedy action in this situation.
Action B is greedy because its current estimated value is greatest.
Exploitation and Exploration
Exploitation means choosing the greedy action: the agent uses its current knowledge to pursue the best expected one-step reward. Exploration means choosing a nongreedy action: the agent gathers information that may improve its estimates later.
| Choice | Action selected | Primary purpose |
|---|---|---|
| Exploitation | Greedy action | Pursue the best expected one-step reward |
| Exploration | Nongreedy action | Gather information that may improve later estimates |
The distinction is based on both the selected action and the purpose of the choice.
Short-Term and Long-Term Outcomes
Exploitation can be the best choice for the next step because it uses the action with the greatest current estimated value. Exploration can be worthwhile when many future selections remain: trying a nongreedy action may reveal that it is better than currently estimated, allowing improved choices later.
A short-term sacrifice for later knowledge
An agent estimates that Action A is better than Action B. It can choose A now for the best expected one-step reward, or choose B to learn more about B. How do the two choices differ?
Exploit now: Choosing A follows the current estimate and aims at the best expected immediate reward.
Explore now: Choosing B is nongreedy under the current estimates, but it creates an opportunity to improve the estimate for B.
Consider future selections: If many future selections remain, information from B may support better choices later and may produce greater total reward.
Exploitation favors the immediate decision; exploration may improve the quality of later decisions.
Common Reasoning Errors
Treating the greedy action as the action that is guaranteed to be best.
A greedy action is defined by having the greatest current estimated value, not by a guarantee about its actual reward.
Fix:
Describe it as the action currently estimated to have the greatest value.Calling every nongreedy choice irrational.
A nongreedy choice can be exploration, which may reveal a better action and improve later estimates.
Fix:
Ask whether the choice is intended to gather information for future decisions.Assuming exploration is always better than exploitation.
Exploration and exploitation serve different purposes and horizons.
Fix:
Separate the immediate-reward goal from the possible long-term value of improved knowledge.Claiming that one action selection both explores and exploits.
Selecting the greedy action uses the current estimate for immediate reward, whereas selecting a nongreedy action creates an opportunity to learn about that action.
Fix:
Classify the selected action as exploitation if it is greedy and exploration if it is nongreedy.
Practice the Decision
An agent has current estimated values of 3 for Action A, 6 for Action B, and 4 for Action C. Identify the greedy action. Then describe what it means to exploit in this situation and what it means to explore instead.
Hints
- Start by finding the greatest current estimated value.
- Exploitation selects the action with that greatest estimate.
- Exploration selects a nongreedy action to gather information that may improve later estimates.
A complete explanation should name the greedy action, connect it to exploitation, and explain that choosing another action would be exploration because it prioritizes information rather than the best expected one-step reward.
Key Takeaways
- A greedy action has the greatest current estimated value.
- Exploitation chooses the greedy action to pursue the best expected one-step reward.
- Exploration chooses a nongreedy action to gather information that may improve later estimates.
- Exploitation can favor immediate reward, while exploration can improve total reward when many future selections remain.
- One action selection cannot simultaneously serve both exploration and exploitation.