Off-policy TD control

Finding Q-learning’s Place

Q-learning is the name of a particular algorithm. The supplied definition places it within reinforcement learning and identifies it more precisely as an off-policy TD control algorithm. Reading that description accurately is the first skill: identify the broad field, the learning approach, the control role, and the policy relationship without adding mechanics that have not yet been introduced.

Reading the Definition Precisely

Two Jobs in TD Control

TD control combines two activities. First, value prediction makes the value function better at predicting returns for the current policy. Second, local policy improvement uses that value information to improve the policy. These activities work together: experience supports value prediction, and the resulting value information supports a local improvement in action preference.

The word control matters because the process is not limited to predicting the returns of a fixed policy. The policy is also improved locally using the value function. Therefore, the on-policy or off-policy label does not say whether prediction or improvement occurs; TD control includes both. The label identifies the relationship between the policy being improved or evaluated and the actions used to produce experience.

Policy Relationships

The key diagnostic question is: does the current policy select the actions used to produce experience? In an on-policy method, the current policy is used to select actions. In an off-policy method, the current policy is not used to select those actions. The policy being improved need not be the policy that selected the observed actions.

Exploration and Learning

An agent must choose actions while it is still learning. The resulting experience helps the value function estimate returns, and that value information can then improve the policy locally. The central design issue is maintaining sufficient exploration while learning from experience. Action choices matter twice: they determine what experience is collected, and they are connected to the policy being evaluated or improved.

Classification Practice

• Treating Q-learning as the name of reinforcement learning as a whole.

  The source identifies Q-learning as a particular algorithm within reinforcement learning.

  Fix: Describe reinforcement learning as the broad field and Q-learning as an algorithm within it.

• Assuming that off-policy means TD control does not perform value prediction.

  TD control combines value prediction for the current policy with local policy improvement.

  Fix: Use off-policy to describe the policy relationship, while retaining both prediction and improvement as parts of TD control.

• Inferring a complete algorithmic expression from the classification alone.

  The supplied material does not provide that defining expression.

  Fix: Keep the conclusion at the supported level: Q-learning belongs to the off-policy TD control category.

• Classifying a method without asking which policy selects actions.

  That relationship is the key diagnostic question in the supplied explanation.

  Fix: Ask whether the current policy is used to select the actions that produce experience.

Key Takeaways

1. Q-learning is an off-policy TD control algorithm in reinforcement learning.
2. TD control combines value prediction for the current policy with local policy improvement.
3. On-policy methods use the current policy to select actions; Sarsa is the supplied example.
4. Off-policy methods do not use the current policy to select actions; Q-learning and Expected Sarsa are the supplied examples.
5. Exploration is central because the agent must collect informative experience while its value function and policy are being improved.