A/B Testing for Machine Learning Systems

Imagine you have built a model that predicts which items a user will want to see next on a shopping website. You have trained it, you have checked it, and it looks promising. The natural question is what kind of checking comes first, what kind comes next, and why the second kind cannot be skipped when real people are involved.

A model that scores well on past data may still change what real people do once it is placed in front of them. Changing the recommendations or search results a website shows can drastically change user behavior, and a change that looks like a clear improvement on a test set can lead to unexpected consequences once people start reacting to it. This article follows the path from the first kind of checking, offline evaluation, toward testing with live users, and looks at one of the most common tools for doing that: A/B testing.

Offline evaluation is a necessary starting point. It is cheap to repeat, it does not expose real users to an unfinished system, and it lets you compare candidate models on identical inputs. Because the test set is fixed in advance, every candidate model is measured against exactly the same examples, which makes the comparison fair.

The limitation is built into the design. The predictions offline evaluation measures are fixed in advance, while the system's behavior in production depends on how people respond to what they are shown. Offline evaluation can tell you how well a model matches the past, but it cannot observe a future in which users react to the model's output.

Once an algorithm is part of a live website, the question changes. It is no longer enough to ask whether the predictions match a test set. The team must ask what happens to the whole system when users encounter the new behavior. This is called online testing, also known as live testing, in which the consequences of employing the algorithm in the overall system are evaluated.

For systems that users see directly, online testing is usually the next step after offline evaluation. Offline evaluation tells you whether a candidate looks promising on saved data; online testing tells you how the candidate behaves inside the product. The diagram below shows the order of these stages.

The procedure has three stages. First, users are divided into two groups, and each group receives one algorithm. Second, relevant success metrics are recorded for both groups over a set period of time. Third, the metrics of algorithm A and algorithm B are compared, and a choice between the two approaches is made according to those metrics. Because users never need to know that a comparison is happening, the study is described as blind.

A common arrangement is for algorithm A to be a new model and algorithm B to be the established system already running. The new model is tested against the current one on live traffic, so the decision to adopt it rests on how users actually behave rather than on offline scores alone. The current system keeps serving everyone outside the tested portion.

The main benefit of A/B testing is that it lets you evaluate algorithms in the wild. Because users are interacting with the model in its real setting, the test can surface effects that no offline measurement would have predicted. This is the protection most user-facing services rely on: a small portion of traffic absorbs the risk of a surprising change before it reaches everyone.

A useful way to hold this idea is to picture the algorithm's output feeding into the rest of the product. Its recommendations are displayed, users click or ignore them, and those reactions shape the metrics the team sees. The effect of an algorithm therefore includes changes in what people do downstream, not only the accuracy of its predictions.

Once the metrics have been recorded, the comparison can be as simple as computing a rate for each group. Suppose the metric is the fraction of visitors who clicked on a recommended item. For each group, the click rate is the number of clicks divided by the number of visitors: $\text{click rate} = \frac{\text{clicks}}{\text{visitors}}$.

A/B testing is not the only way to run online tests. There are more elaborate mechanisms that go beyond A/B testing, and bandit algorithms are one example. They belong to the same online testing family, so they also evaluate an algorithm's consequences with live use, but they are more involved than a simple split between two fixed groups.

The source does not describe how bandit algorithms work internally, and it points to a dedicated book on the subject for a proper introduction. For now, the key point is that A/B testing is a well-established starting approach within a larger set of options. You do not need to master bandit algorithms to understand live evaluation, but it helps to know they exist for cases where a simple two-group comparison is not the right fit.