Bandit Algorithms for Online Testing
Algorithm A is often a new model while B is the established system, and bandit algorithms are a more elaborate online testing mechanism beyond A/B testing.
Why a Good Test Score Is Not Enough
Suppose you have built a model that recommends products on a shopping website. You check it against a set of examples and it looks strong. Should you assume it will perform just as well when real shoppers see its recommendations? The answer depends on what you are trying to learn, and this article explains why a strong test result is only the beginning when a machine learning system faces users.
What do you think happens?
If a recommendation method wins on a test set collected beforehand, will it always win once it is placed inside a live product?
Reveal answer
Not necessarily. Predictions that look good on a test set can still behave differently once they are placed inside a real product.
A test set only checks whether predictions match what happened in the collected examples. Once users see the recommendations, their clicks, purchases, and time spent can change, and those changes can ripple through the rest of the system.
Offline Evaluation: Checking Predictions
Offline evaluation checks an algorithm's predictions against a test set collected beforehand. It is the first step for a user-facing machine learning system.
Much of machine learning evaluation begins with a test set. You collect a set of examples ahead of time, let your algorithm make predictions on them, and measure how well those predictions match what actually happened. This process lets you check an algorithm's predictions without exposing anything to real users. Nothing the algorithm does is shown to a customer, so the test carries no risk to the live product.
Offline evaluation answers one specific question: how well do this algorithm's predictions match the test set? That is a valuable question, but it does not ask what happens when the algorithm is part of a product that people actually use.
Online Testing: Measuring Consequences
If your machine learning system is user-facing, offline evaluation is only the first step. The next step is usually online testing, also called live testing. Online testing evaluates the consequences of employing an algorithm in the overall system, not just the accuracy of its predictions.
Imagine a website that shows a list of recommended articles. Offline, the new recommendation list scores well against historical data. Online, the list is shown to readers, and the readers start clicking different items, staying longer on some pages, and leaving sooner from others. None of these shifts appears in the offline test, because the offline test never showed anything to a reader.
This is the central reason online testing exists. Changing the recommendations or search results a website shows its users can drastically change how those users behave. A recommendation list that performs well offline may shift what people click, what they buy, or how long they stay, and those shifts can ripple through the rest of the system in ways no one predicted. Online testing is designed to discover these unexpected consequences when users interact with a model.
How A/B Testing Splits Users
A/B testing is a form of blind user study. A selected portion of users receives a website or service that uses algorithm A, while the rest receive the same kind of service using algorithm B. Success metrics are recorded for both groups over a set period, and the two groups are then compared.
The word blind matters. Without their knowledge, users are assigned to one version or the other. Because users do not know which version they see, their behavior reflects what happens in ordinary use rather than how they behave when they know they are being tested. This is what makes the comparison meaningful for a live product.
In a typical setup, algorithm A is often a new model, while algorithm B is the established system. The new model is tested against the current one so that the team can see whether it improves the metrics they care about in real use.
Tracing One Test Through the Numbers
Comparing two recommendation lists
A news site runs an A/B test on its article recommendations. Algorithm A is the new model and algorithm B is the current system. Each group has 500 readers over one week. Clicks on recommended articles are the success metric. Algorithm A produced 60 clicks and algorithm B produced 45 clicks. Which algorithm performed better on this metric?
- 1Assign readers to groups
Readers are split into two groups without their knowledge. One group receives algorithm A and the other receives algorithm B. Each group has 500 readers.
- 2Record the success metric
Clicks on recommended articles are counted for each group over the set week.
- 3Convert counts to rates
Algorithm A: 60 clicks out of 500 readers gives 12 percent. Algorithm B: 45 clicks out of 500 readers gives 9 percent. Rates are used because the group sizes are equal here, but rates keep the comparison fair whenever group sizes differ.
- 4Compare and choose
Algorithm A has the higher click rate on this metric, so the team has evidence in favor of A. The decision is made according to the recorded metrics, as the A/B method describes.
Algorithm A produced a 12 percent click rate against 9 percent for algorithm B over the week. On this metric, A performed better in this test.
Watching Traffic Shift in Bandit Testing
A/B testing is not the only way to run online tests. There are more elaborate mechanisms for online testing that go beyond A/B testing, and bandit algorithms are one example. The source introduces them at a high level only: the mechanics of how they choose between options are a topic for further study. What matters at this stage is recognizing that they exist as a more elaborate option.
The contrast to keep in mind is this. An A/B test keeps its two groups fixed for the whole test period. A bandit approach is described as adaptive, meaning the share of traffic sent to each option can change as results come in. The illustration below is generated to show that idea in outline. It is not a description of any specific bandit method.
| Feature | A/B testing | Bandit approach |
|---|---|---|
| Assignment of users | Two groups, one for algorithm A and one for algorithm B | Traffic is allocated across options and the allocation can change |
| Traffic over the test | Fixed split for the set period | Adaptive, can shift toward an option as results accumulate |
| Position in online testing | A common online testing mechanism | A more elaborate mechanism beyond A/B testing |
| Measures | Success metrics compared for both groups | Online testing of consequences in the overall system |
A/B testing and bandit approaches as described in the source, at the level of structure only.
Mistakes When Reading Online Test Results
Common mistake Assuming a strong offline score guarantees a good live result
What it looks likeA recommendation list wins on a test set, so the team ships it without a live test.
Why it failsPredictions that look good on a test set can still behave differently once placed inside a real product.
Do this insteadUse offline evaluation as the first step, then run online testing to observe the consequences for users.
Common mistake Thinking an A/B test reveals how users behave when they know they are being tested
Why it failsA/B testing is a blind user study. Users do not know which version they see, so their behavior reflects ordinary use.
Do this insteadRecognize that the blindness is intentional and is what makes the measured behavior realistic.
Common mistake Judging a winner before the set period ends
Why it failsThe source describes comparing metrics for both groups after a set period of time has passed.
Do this insteadRecord metrics over the full set period before comparing algorithm A and algorithm B.
Common mistake Treating bandit algorithms as the same thing as a fixed A/B split
Why it failsBandit algorithms are described as a more elaborate mechanism for online testing, beyond A/B testing.
Do this insteadKeep the two separate in your mind: A/B testing uses fixed groups, while bandit approaches are adaptive.
Practice: Classify the Testing Stage
A team has a new recommendation model. Describe for each step whether it belongs to offline evaluation or online testing, and justify your choice in one sentence: (1) checking predictions against a test set collected beforehand; (2) placing the model in the live site and recording how readers behave; (3) splitting readers into two groups that receive different algorithms and comparing success metrics after a set period.
Hints
- Ask whether the step uses a test set collected beforehand or real users interacting with the product.
- Steps that show the algorithm to real users and record consequences belong to online testing.
- Step 3 describes a form of online testing; think about who receives each algorithm.
Key Takeaways
- Offline evaluation checks an algorithm's predictions against a test set collected beforehand, without exposing anything to real users.
- Online testing, also called live testing, evaluates the consequences of employing an algorithm in the overall system, which can reveal effects that a test set cannot.
- A/B testing is a blind user study: a selected portion of users receives algorithm A and the rest receive algorithm B, and success metrics are compared over a set period.
- Algorithm A is often a new model while B is the established system, and bandit algorithms are a more elaborate online testing mechanism beyond A/B testing.