Constant-α Action-Value Methods
Initial estimates affect action-value methods because they provide the starting values used before experience is available.
Before Experience Exists
An action-value method must start with an estimate for every action, even before any action has produced experience. These starting values are called initial estimates. They are not neutral: before rewards are available, they can influence which actions the method selects and can continue to affect later estimates.
Bias from initial estimates is the continuing influence of the starting action-value estimates on a method's behavior and later estimates. The key question is not whether the starting values matter at the beginning, but how long their influence remains.
What do you think happens?
Suppose some actions have not yet been selected. Has the initial-estimate bias necessarily disappeared?
Reveal answer
Answer: No, because the condition for disappearance has not been met.
The source states that sample-average methods lose this bias after all actions have been selected at least once. If some actions have not been selected, that condition is not satisfied.
Sample-Average Updates
Sample-average methods combine the rewards observed for an action over time. Their initial estimate can affect behavior while experience is still limited, but the source identifies a clear point at which this initial-estimate bias is lost: after all actions have been selected at least once. At that point, each action has experience rather than only a starting value, so the initial estimate no longer contributes to the method's action-value estimate in the stated sense.
A Sample-Average Timeline
Track the role of an initial estimate when a method has not yet selected every action.
Start: Every action has a starting action-value estimate because no experience is available yet. Those estimates can influence early behavior.
Partial experience: Some actions have now been selected, but at least one action has not. The source's condition for losing initial-estimate bias has not yet been met.
Complete first coverage: Once all actions have been selected at least once, the sample-average method loses the initial-estimate bias described in the source.
For sample-average methods, the initial estimates can matter during the early phase, but their bias is lost after every action has been selected at least once.
Constant-α Updates
Methods with a constant α behave differently. Repeated experience weakens the effect of the initial estimate, so later estimates depend less strongly on where the method started. However, the source describes this influence as permanent. In this context, decreasing does not mean eventually erased. It means that the starting value matters less over time while some dependence on it remains.
A Constant-α Timeline
Compare the role of a starting estimate across repeated constant-α updates.
Start: The action-value estimate begins at the user's chosen initial value. Before experience is available, this value can affect behavior.
Early updates: New experience changes the estimate, but the starting value still contributes to the method's later estimate.
Later updates: Further experience reduces the relative influence of the starting value. The source nevertheless describes that influence as permanent rather than erased.
A constant-α method gradually weakens, but does not eliminate, the influence of its initial action-value estimates.
| Method | Early role of initial estimates | Longer-term influence |
|---|---|---|
| Sample-average | Can influence behavior before enough experience is available | Bias is lost after all actions have been selected at least once |
| Constant α | Can influence behavior before enough experience is available | Influence decreases but remains permanent |
Prior Knowledge and Parameter Choice
Choosing initial estimates is an extra decision made by the user. The choice might be simple, such as assigning the same starting value to every action, but it is still a starting assumption. Because constant-α methods retain some influence from those values, an arbitrary choice can create unwanted dependence on that assumption.
| Use of initial estimates | What it means | Practical interpretation |
|---|---|---|
| Prior knowledge | The starting values express beliefs about expected rewards before experience is available | The bias is deliberate and can be useful |
| Extra parameter | The starting values are chosen without intended prior knowledge | The method has an additional assumption that must be chosen |
Suppose a practitioner has prior knowledge about the expected rewards of several actions before running the method. Choosing initial estimates to reflect that knowledge can make the starting bias intentional rather than accidental. If the practitioner simply chooses initial values without such knowledge, those values become an extra parameter that may introduce unwanted dependence on a starting assumption.
Mistakes to Avoid
Assuming that initial-estimate bias disappears as soon as one action has been selected.
The source states that sample-average bias is lost after all actions have been selected at least once.
Fix:
Check whether every action has been selected before concluding that sample-average initial-estimate bias has disappeared.Treating decreasing influence under constant α as complete disappearance.
The source describes constant-α bias as permanent but decreasing.
Fix:
Say that the influence has been reduced, not erased.Assuming that choosing initial estimates to be simple makes them neutral.
Choosing initial estimates is still a user decision, and the source identifies that decision as a possible source of unwanted dependence.
Fix:
Recognize the chosen starting values as an additional parameter, even when they are all set alike.Treating every initial estimate as an unwanted artifact.
The source states that initial estimates can deliberately express prior knowledge.
Fix:
Distinguish intentional prior knowledge from an arbitrary starting assumption.
Check Your Understanding
A method has selected some, but not all, available actions. Explain whether sample-average initial-estimate bias has disappeared. Then explain what can be said about the initial-estimate influence under a constant-α method after repeated updates.
Hints
- Recall the condition involving every action being selected at least once.
- For constant α, distinguish a decreasing influence from an erased influence.
Decide whether each situation describes useful prior knowledge or merely an extra parameter choice: initial estimates are chosen to reflect beliefs about expected rewards; initial estimates are chosen arbitrarily because the method requires starting values.
Hints
- Ask whether the values intentionally represent beliefs formed before experience.
- If they do not, treat them as a user-selected starting assumption.
Key Takeaways
- Initial estimates are the starting action-value estimates used before experience is available, and they can influence early behavior.
- For sample-average methods, initial-estimate bias is lost after all actions have been selected at least once.
- For constant-α methods, experience reduces the influence of initial estimates, but the source describes that influence as permanent.
- Choosing initial estimates is an extra user decision that can create unwanted dependence on a starting assumption.
- The same choice can be useful when the initial estimates intentionally express prior knowledge about expected rewards.
Key Takeaways
- Initial action-value estimates matter because they guide the method before experience is available.
- Sample-average methods lose this bias after every action has been selected at least once.
- Constant-α methods reduce, but do not erase, the influence of their initial estimates.
- Initial estimates are both a possible source of unwanted bias and a tool for expressing prior knowledge.