Concepts / Action-Value Methods for Decision Making

Action-Value Methods for Decision Making

Greedy action selection chooses an action with the highest estimated action value.

  • Programming

Choosing by Estimated Value

When a decision-making method has estimated action values, it needs a rule for choosing an action. Greedy action selection uses the simplest rule: compare the estimated values and choose an action with the highest estimated value. The decision depends on the ordering of the estimates: the action selected is one that reaches the maximum.

comparehighestcompareAction Aestimated value 4Action Bmaximum value 7Action Bestimated value 7Action Cestimated value 5
How do the estimated values of several actions compare, and which action does the greedy rule select?

Tracing the Maximum

Selecting One Greedy Action

Suppose three available actions have these estimated values: Action A has 4, Action B has 7, and Action C has 5. Which action does greedy selection choose?

Compare: Place the estimated values in order from lowest to highest: 4, 5, and 7.

Find the maximum: The highest estimated value is 7.

Map the maximum to an action: The value 7 belongs to Action B, so Action B reaches the maximum.

Select: The greedy rule selects Action B because it has the highest estimated action value.

Action B is selected.

Action A4Action B7Action C5
What estimated value is associated with each available action?
not maximummaximumnot maximum4Action AAction Bselected7Action B5Action C
How does finding the maximum value map to the action that is selected?

The Greedy Rule

Greedy action selection chooses an action with the highest estimated action value.

The rule can be written as Qₜ(A*ₜ) = maxₐ Qₜ(a). Here, the estimated value associated with the selected action equals the maximum estimated value among the available actions. To apply the rule, compare the estimates, find the maximum, and select an action that reaches it.

  • List the available actions and their estimated action values.
  • Compare the estimated values.
  • Find the maximum estimated value.
  • Select an action whose estimated value reaches that maximum.

Resolving Equal Maxima

A maximum does not always belong to only one action. If two or more actions share the highest estimated value, they are all greedy actions. The greedy rule allows selecting one of the actions tied at the maximum.

A Tie at the Maximum

Suppose Action A has an estimated value of 6, Action B has 6, and Action C has 3. Which actions can greedy selection choose?

Compare: The estimates are 6, 6, and 3.

Find the maximum: The highest estimated value is 6.

Identify every action reaching it: Both Action A and Action B have the maximum value of 6.

Select: Greedy selection can choose Action A or Action B because both are tied at the maximum.

The possible greedy actions are Action A and Action B.

maximummaximumbelow maximumAction Aestimated value 6Action A or Action Bgreedy selectionAction Bestimated value 6Action Cestimated value 3
What happens when two or more actions share the maximum estimated value, and which actions can be selected?

Common Selection Mistakes

  • Selecting the action with the lowest estimated value.

    Greedy selection chooses an action with the highest estimated action value.

    Fix: Compare the estimates and select an action that reaches the maximum.

  • Stopping after finding one maximum without checking for ties.

    A tie at the maximum creates multiple greedy actions.

    Fix: Identify every action whose estimated value equals the maximum.

  • Treating the action name as the selection criterion.

    The selected action is determined by the ordering of the estimated values.

    Fix: Use the numerical estimates to locate the maximum.

  • Assigning an unspecified behavior to another action-selection method.

    The greedy rule is defined here, but the source does not specify the details of other action-selection methods.

    Fix: Describe only the greedy criterion unless another method has been defined separately.

Apply the Selection Rule

EASY

Three actions have estimated values: Action A = 8, Action B = 5, and Action C = 8. Identify the maximum estimated value and list every action that greedy selection can choose.

Hints
  • First compare 8, 5, and 8.
  • Then identify every action associated with the maximum.

What do you think happens?

Which actions are eligible under the greedy rule when Action A = 8, Action B = 5, and Action C = 8?

  • Action A only
  • Action B only
  • Action C only
  • Action A or Action C
Reveal answer

Answer: Action A or Action C

The maximum estimated value is 8, and both Action A and Action C reach it. A tie at the maximum creates multiple greedy actions.

Selection Checklist

  1. Greedy action selection chooses an action with the highest estimated action value.
  2. The maximum estimated value determines which actions are eligible.
  3. With one maximum, the action associated with it is the greedy selection.
  4. With a tie, every action at the maximum is a greedy action, and one of them may be selected.
  5. The reliable procedure is compare, find the maximum, and identify all actions that reach it.

Key Takeaways

  • Greedy selection uses the highest estimated action value as its decision criterion.
  • The selected action is found by mapping the maximum estimate back to its action.
  • When multiple actions share the maximum, each is a possible greedy action.
  • Always compare all estimates and check for ties before identifying the selection.

Related Topics