ε-Greedy Action Selection
High initial action-value estimates can make a greedy method explore without explicitly using ε-greedy random selection.
The Exploration Dilemma
When an agent must choose an action, it can use what it currently believes about the available actions or choose an action that may teach it more. The action with the greatest current estimated value is the greedy action. Selecting it is exploitation: the agent uses its current knowledge to seek the best expected one-step reward. Selecting a nongreedy action is exploration: the agent gathers information that may improve its estimates later.
Greedy Actions and ε-Greedy Choice
A greedy action is an action with the greatest current estimated value. A greedy selection exploits the agent's present knowledge rather than deliberately selecting a nongreedy action to gather more information.
A single action selection cannot fully serve both purposes at once. If the agent selects the greedy action, it uses its current estimate to pursue immediate reward. If it selects a nongreedy action, it creates an opportunity to learn more about that action and possibly improve later decisions. This conflict is the reason action selection must balance exploitation and exploration.
Comparing estimated action values
An agent currently estimates the values of three actions as A = 4, B = 7, and C = 5. Which action is greedy, and what do exploitation and exploration mean in this situation?
Find the greatest estimate: The greatest current estimated value is 7, which belongs to action B.
Identify the greedy action: Action B is the greedy action because it has the greatest current estimated value.
Interpret exploitation: Choosing B exploits current knowledge and seeks the best expected one-step reward according to the estimates.
Interpret exploration: Choosing A or C instead would be exploration because either choice is nongreedy and may provide information that improves later estimates.
B is greedy. B represents exploitation; A and C represent possible exploratory choices.
Optimistic Estimates in Action
A greedy method does not necessarily remain focused on one action forever. If the initial action-value estimates are deliberately high, an action that has not yet produced enough evidence can look attractive. The greedy method then selects it because its current estimate is high. After the agent gains experience and updates its estimates, actions that are not supported by the observed rewards become less attractive. This causes exploration to decrease over time even though the selection rule remains greedy.
A greedy method that explores
Suppose three actions all begin with estimates that are higher than the rewards the agent has reason to expect. The agent always chooses the action with the greatest current estimate.
Initial selection: The greedy rule selects whichever action currently has the greatest optimistic estimate. This is exploitation according to the current estimates, even though the high estimates cause the method to try actions it has not yet evaluated well.
Experience changes the estimates: After the selected action produces experience, its estimate is updated. The relative ordering of the action estimates can change.
Further selections: Another action may now have the greatest estimate, so the same greedy rule selects that action. The method has explored multiple actions without explicitly using random ε-greedy selection.
Exploration fades: As the initial optimism is reduced by experience, fewer untested actions remain artificially attractive. The extra exploration therefore decreases with time.
Optimistic initial estimates can make greedy selection explore early, but the exploration effect is temporary.
Early Cost and Later Benefit
Optimistic initial values can hurt early performance because the agent spends selections testing actions whose high estimates are not yet justified. Those selections may produce less reward than immediately choosing the action that currently appears best under more realistic estimates. The same exploration can support better later performance: by trying alternatives, the agent may discover a better action and improve its future choices.
| Method | How exploration occurs | Early performance | Later behavior |
|---|---|---|---|
| Greedy with optimistic initial values | High initial estimates make different actions appear attractive | May be worse because the agent tests alternatives | Exploration decreases as estimates are updated |
| ε-greedy | The method deliberately selects a nongreedy action with probability ε | Depends on the balance between greedy and nongreedy selections | The exploration mechanism is explicit rather than only an initial effect |
The comparison does not support the rule that more exploration must always produce better performance. A greedy method with optimistic initial action-value estimates can explore heavily at the beginning, perform worse initially, and then perform better later as its exploration decreases. ε-greedy selection makes the exploration choice explicit by allowing a nongreedy action with probability ε. The methods therefore differ in when and why they explore.
Stationary and Changing Problems
The benefit of optimistic initial values depends on the problem being stationary. In a stationary problem, the action values do not change in the relevant sense over time, so information gathered during the initial exploration can remain useful. In a nonstationary problem, action values change over time. Exploration caused only by initial optimism is temporary, so it may stop providing useful adaptation after the initial estimates have been updated.
Common Reasoning Errors
Assuming that greedy selection can never explore.
Different actions can become the current maximum as experience updates the estimates, so a greedy method can explore without explicit random selection.
Fix:
Ask why the current estimates change before deciding whether a greedy rule will repeatedly choose the same action.Assuming that more exploration must always produce better results.
Extra exploration can reduce early rewards, and its later benefit depends on the problem and the number of future selections.
Fix:
Separate immediate performance from later performance and consider whether the problem is stationary.Treating exploitation as always correct.
The current estimate may be incomplete, and a nongreedy action may reveal information that supports better future choices.
Fix:
Recognize exploitation as a choice for immediate reward, not a guarantee of the greatest total reward.Treating optimistic initial values as continuing exploration forever.
The exploration effect is temporary and decreases as the initial optimism is removed through experience.
Fix:
Track whether the method has an ongoing exploration mechanism or only an initial source of exploration.
Reasoning Practice
An agent estimates action P at 8, action Q at 6, and action R at 5. It can either choose the greedy action or deliberately choose a nongreedy action to learn more. Explain what the two choices represent, which choice is likely to favor immediate reward according to the current estimates, and why the nongreedy choice could still improve later decisions.
Hints
- Identify the action with the greatest current estimated value.
- Use exploitation for the greedy choice and exploration for the nongreedy choice.
- Distinguish the next reward from the value of information for future selections.
Compare these two situations: a greedy method with optimistic initial estimates in a stationary problem, and the same method in a nonstationary problem. Explain why initial exploration may help in the first situation but may not be a sufficient exploration strategy in the second.
Hints
- Ask whether information gathered early remains relevant.
- Recall that the exploration caused by optimism decreases after estimates are updated.
- Connect changing action values with the need for useful information beyond the initial phase.
Key Takeaways
- A greedy action has the greatest current estimated value.
- Exploitation selects the greedy action for the best expected one-step reward; exploration selects a nongreedy action to gather information for later choices.
- Optimistic initial estimates can make a greedy method explore early because untested actions appear highly valuable.
- Optimistic exploration may reduce early reward but support better later performance as the agent discovers better actions and updates its estimates.
- The benefit depends on stationarity; because the effect is temporary, optimistic initial values are not a general exploration solution for nonstationary tasks.
Key Takeaways
- The greedy action is the action with the greatest current estimated value.
- Exploitation targets immediate reward, while exploration may improve total reward across future selections.
- Optimistic initial estimates can cause a greedy method to explore without explicitly selecting nongreedy actions at random.
- This optimism can cost reward initially and provide benefits later, especially in stationary problems.
- Because initial optimism creates only temporary exploration, it is not a generally suitable method for nonstationary tasks.