Sample-Average Action-Value Methods
Initial estimates affect action-value methods because they provide the starting values used before experience is available.
Why Starting Values Matter
An action-value method must have an estimate for an action before that action has produced any experience. That starting value is called an initial action-value estimate. It is not necessarily neutral: before enough experience has been collected, it can influence which actions the method selects and therefore influence later behavior.
Initial-estimate bias is the influence that starting action-value estimates have on an action-value method's behavior and later estimates before experience has removed or reduced that influence.
The useful question is not whether initial estimates matter at all. They do matter at the beginning. The important question is how long their influence remains under the update method being used.
Sample Averages Forget the Start
A sample-average method updates an action's estimate from the rewards observed for that action. Each new experience contributes to the estimate, and the estimate is based on the observed sample rather than continuing to preserve the initial value as a separate influence.
Tracing One Action
Suppose one action begins with an initial estimate, then produces experience. How does a sample-average method's dependence on that initial estimate change?
Before selection: The action has no experience yet, so the method needs its initial action-value estimate.
After the first selection: The action now has experience. The observed reward contributes to its sample-based estimate.
After further selections: More observed rewards determine the estimate, while the initial estimate no longer remains as a continuing source of bias in the sample-average method.
After every action has been selected: The condition stated by the source for losing the initial-estimate bias has been met: all actions have been selected at least once.
Sample-average methods lose their initial-estimate bias after all actions have been selected at least once.
Constant-α Remembers Gradually
A method using a constant α also allows experience to weaken the effect of the initial estimate. However, the source distinguishes this behavior from the sample-average case: the influence decreases but is not eventually erased. Later estimates continue to retain some dependence on the starting values.
Repeated Constant-α Updates
An action begins with an initial estimate and is updated repeatedly with a constant α. What happens to the initial estimate's influence?
First update: The new experience changes the current estimate, but the initial estimate still contributes to the result.
Later updates: Additional experience reduces the relative influence of the starting value.
Longer run: The influence becomes smaller, but the source describes it as permanent rather than erased.
Constant-α methods retain a decreasing influence from their initial estimates: reduced does not mean gone.
Two Ways to Weight Experience
The central difference is how each method treats the starting value as experience accumulates. A sample-average method eventually loses its initial-estimate bias once every action has been selected at least once. A constant-α method lets later experience reduce the influence of the starting value, but continues to retain some dependence on it.
| Method | Early behavior | Longer-term influence of initial estimates | Key condition |
|---|---|---|---|
| Sample-average | Uses initial estimates before experience is available | Initial-estimate bias is lost | All actions have been selected at least once |
| Constant-α | Uses initial estimates before experience is available | Influence decreases but remains | Repeated updates reduce, but do not erase, dependence |
Qualitative comparison of initial-estimate influence
Prior Knowledge or Extra Parameter
Choosing initial estimates is a user decision. Setting every initial estimate to zero is still a choice, and it can introduce dependence on a starting assumption. In that sense, initial estimates are an extra parameter to choose.
The same choice can also be useful. If there is prior knowledge about expected rewards before experience is available, initial estimates can represent that knowledge. Initial estimates are therefore both a possible source of bias and a tool for expressing beliefs about expected rewards.
Treat an initial estimate deliberately. Use it to express prior knowledge when that knowledge is meaningful. Otherwise, remember that the selected starting value is an additional assumption that may affect early behavior.
Mistakes About Bias
Assuming an initial estimate is neutral simply because it is used for every action.
The source states that choosing initial estimates is a user decision and that starting values can influence later behavior.
Fix:
Treat even a uniform starting choice as an assumption that can affect early action selection.Claiming that sample-average methods lose the bias immediately after one action is selected.
The source gives a condition involving all actions, not just one action.
Fix:
Say that the bias is lost after all actions have been selected at least once.Saying that constant-α updates eventually erase the initial estimate.
The source describes the influence under constant α as permanent but decreasing.
Fix:
Describe the influence as reduced, not gone.Treating initial estimates only as a problem.
The source says initial estimates can deliberately represent prior knowledge.
Fix:
Recognize both roles: initial estimates can cause bias and can encode useful prior knowledge.
Check Your Understanding
A method has selected some, but not all, available actions. First, decide whether a sample-average method has definitely lost its initial-estimate bias. Then decide whether a constant-α method's initial-estimate influence is gone or merely reduced. Finally, explain whether choosing initial estimates can ever be useful.
Hints
- Use the condition involving every available action.
- Separate decreasing influence from disappearing influence.
- Think about prior knowledge of expected rewards.
What do you think happens?
If one available action has never been selected, has a sample-average method already lost all initial-estimate bias?
Reveal answer
Answer: No, because all actions must have been selected at least once.
The source states that sample-average methods lose this bias after all actions have been selected at least once.
What do you think happens?
After repeated constant-α updates, what happens to the influence of the initial estimate?
Reveal answer
Answer: It decreases but remains.
The source describes constant-α bias as permanent but decreasing. Later experience weakens the dependence without erasing it.
Key Takeaways
- Initial action-value estimates provide the values used before an action has produced experience, so they can influence early behavior.
- This influence is called bias from the initial estimates.
- Sample-average methods lose this bias after all actions have been selected at least once.
- Constant-α methods reduce the influence of initial estimates over time but retain some dependence on them.
- Initial estimates are an extra user choice, but they can also encode useful prior knowledge about expected rewards.
Key Takeaways
- Initial estimates matter because action-value methods need starting values before experience is available.
- Sample-average methods eventually lose their initial-estimate bias once every action has been selected at least once.
- Constant-α methods retain a decreasing, permanent influence from their initial estimates.
- Choosing initial estimates can add a parameter to the method or deliberately express prior knowledge.