Statistical Learning Theory

Acting While Learning

A bandit problem describes a difficult decision pattern: an agent must act, learn from what happens, and then use that information when making later decisions. The challenge is that action and learning are connected. A decision is not only a way to produce an outcome; it can also provide information that changes what the agent knows.

This is why bandit problems have been studied across several fields. Statistics treats them as sequential design of experiments. Engineering describes the tension as simultaneous identification and control, also called dual control. Psychology uses bandit problems in statistical learning theory, while heuristic search literature often uses the term greedy. These perspectives are different ways of examining the same pattern: acting while still learning.

The Sequential Experiment

The statistical perspective focuses on sequential design of experiments. An experiment is sequential when decisions are made in an order and later decisions can depend on information obtained earlier. In a bandit problem, selecting an option is therefore both a decision and an experiment: the result supplies information that may affect what is selected next.

Exploration and Exploitation

Exploration seeks information. It means taking an action partly because the result may help identify which option is better. Exploitation uses current knowledge. It means acting according to what is currently believed to be the best option. In a sequential setting, these goals interact because an exploratory action can change later knowledge, while an exploitative action uses knowledge already obtained.

Engineering uses a closely related vocabulary. The tension between learning about options and using current knowledge is described as a conflict between identification and control. This is also called dual control. Identification corresponds to seeking information, while control corresponds to using what is currently known to guide action.

Information Across Decisions

The word sequential matters because information does not remain isolated in the decision that produced it. A result obtained from one action can affect the information available at the next step. That changed information can then influence the next choice, creating a chain of decisions linked by learning.

A useful way to read this process is to separate the immediate result from the informational consequence. The immediate result belongs to the action just taken. The informational consequence can persist into later decisions by changing what the agent currently knows. This is the mechanism that makes a bandit problem sequential rather than a collection of unrelated choices.

Four Disciplinary Lenses

Statistics emphasizes the ordered design of experiments and the way later choices depend on earlier information. Engineering emphasizes the tension between identification and control. Psychology places bandit problems within statistical learning theory. Heuristic search literature often uses the term greedy, highlighting the use of a currently preferred choice. These are not unrelated subjects; they emphasize different parts of the same decision pattern.

Mistakes About Sequential Learning

• Treating exploration and exploitation as unrelated activities

  In a sequential decision problem, an action can both produce an outcome and provide information for later choices.

  Fix: Ask what the action accomplishes immediately and what it reveals for subsequent decisions.

• Ignoring the word sequential

  Later decisions can depend on information obtained earlier.

  Fix: Trace the path from an earlier action to its outcome, to updated knowledge, and then to a later choice.

• Equating exploitation with complete certainty

  Exploitation uses current knowledge; it does not remove the need to learn.

  Fix: Describe exploitation as acting on the option currently believed to be best.

• Treating the disciplinary terms as unrelated topics

  Each perspective emphasizes a different part of acting while still learning.

  Fix: Connect the terms through the shared tension between gathering information and using current knowledge.

Practice the Trace

Historical Perspective

The sequential-design perspective has a documented history. Thompson introduced this perspective in 1933 and 1934, Robbins contributed to it in 1952, Bellman studied the area in 1956, and Berry and Fristedt provided an extensive statistical treatment in 1985.