Action Preferences and Softmax Action Probabilities

Reward Feedback Starts the Update

A gradient bandit does not simply count wins and losses for each action. Instead, it maintains action preferences. Those preferences influence the probabilities used to select actions. Stochastic gradient ascent changes the preferences only after two events have occurred: an action has been selected and its reward has been observed.

The Moving Reward Baseline

The latest reward is judged relative to a baseline rather than in isolation. That baseline is the average of all rewards received up through and including the current time t. Because it is computed incrementally, the baseline changes as new rewards arrive. It is therefore a moving point of comparison, not a fixed target selected once at the beginning.

Preference Changes Redistribute Probability

Preferences are not the same thing as stored win counts. They are values that influence the probabilities of selecting actions through a softmax action-selection process. When feedback favors the selected action, its preference moves in the favorable direction and the non-selected preferences move oppositely. The resulting softmax probabilities redistribute toward the selected action. When feedback discourages the selected action, the selected action moves in the unfavorable direction and the probability redistribution goes the other way.

Selected and Non-Selected Actions

The selected action and the non-selected actions are updated in opposite directions because the feedback is used to favor or discourage the action that produced the observed reward while changing the relative standing of the alternatives. Above-baseline feedback favors the selected action, so the non-selected actions move oppositely. Below-baseline feedback discourages the selected action, so the non-selected actions move in the reverse direction relative to it.

The Step-Size Parameter α

The symbol α denotes the positive step-size parameter. It controls the size of a preference adjustment. The reward-baseline comparison determines the qualitative direction: above-baseline feedback favors the selected action, while below-baseline feedback discourages it. α determines how strongly the update is applied. The supplied material establishes the direction and the role of α, but not an exact numerical update amount; that amount also depends on the complete update rule and the action-selection probabilities.

Common Reasoning Errors

• Treating the latest reward as an absolute judgment

  The reward is judged relative to the incremental average of rewards through the current time, not by its size alone.

  Fix: Compare the current reward with the running reward average before deciding the direction.

• Thinking preferences are updated when an action is selected

  Stochastic gradient ascent updates preferences after an action has been selected and its reward has been observed.

  Fix: Wait for reward feedback before describing a preference update.

• Updating only the selected action

  The non-selected actions move oppositely, contributing to the redistribution of action probabilities.

  Fix: Describe both sides of the update: the selected action and the non-selected alternatives.

• Interpreting the algorithm as a simple win-loss counter

  The gradient bandit maintains preferences that influence action probabilities rather than storing only simple counts of wins and losses.

  Fix: Reason in terms of preferences, reward feedback, the baseline, and softmax probabilities.

• Ignoring α when discussing update size

  α is the positive step-size parameter, and the exact update amount also depends on the complete update rule and action-selection probabilities.

  Fix: Use α to discuss the scale of the adjustment, while avoiding unsupported numerical values.

Check Your Prediction

Working Mental Model

1. Preferences are updated only after an action has been selected and its reward has been observed.
2. The current reward is compared with an incremental baseline equal to the average of rewards through the current time.
3. Above-baseline feedback favors the selected action; below-baseline feedback discourages it.
4. Non-selected actions move oppositely, allowing the softmax action probabilities to redistribute.
5. α is the positive step-size parameter that controls the magnitude of the preference adjustment, while the complete update rule determines the exact amount.