A/B Testing in Production

Imagine you have built two versions of a recommendation system. One performs well when you check its predictions against a set of examples you already collected. The question is whether that good score is enough to release it to every user of your website. This article explains why the answer is usually no, and what a team does next.

Much of machine learning evaluation begins with a test set. You collect a set of examples ahead of time, let your algorithm make predictions on them, and measure how well those predictions match what actually happened. This process is called offline evaluation. Its main advantage is that it lets you check an algorithm's predictions without exposing anything to real users.

Offline evaluation is therefore a controlled check. The examples were gathered before the test, the algorithm's outputs are compared against outcomes that already happened, and no live customer sees a different experience while this is going on.

For a user-facing system, offline evaluation is only the first step. The two stages answer different questions. Offline evaluation asks how well an algorithm's predictions match a test set collected beforehand. Online testing, also called live testing, asks what happens when that algorithm is actually part of the product and real users interact with it. Online testing evaluates the consequences of employing an algorithm in the overall system, not just the accuracy of its predictions.

Changing the recommendations or search results a website shows its users can drastically change how those users behave. A recommendation list that performs well in an offline test may shift what people click, what they buy, or how long they stay. Those shifts can ripple through the rest of the system in ways no one predicted.

This is the central reason online testing exists. It is designed to discover unexpected consequences when users interact with a model. A test set cannot show these effects, because the test set does not respond to what the system shows. Real users do.

Read the loop from the top. The system shows recommendations, users respond, and those responses become part of what the system sees next. Because the change in behavior is produced by the algorithm itself, its effects can accumulate in ways a single offline check would never capture.

A/B testing is a form of blind user study. Without their knowledge, a selected portion of users is given a website or service that uses algorithm A, while the rest of the users are given the same kind of service using algorithm B. Because users do not know which version they see, their behavior reflects what happens in ordinary use rather than how they behave when they know they are being tested.

In many settings, algorithm A is a new model and algorithm B is the established system. The established system acts as the baseline that the new model must be compared against.

Follow the flow in the diagram. Incoming users reach the test, and a selected portion is routed to group A while the rest go to group B. For both groups, relevant success metrics are recorded over a set period of time. When that period ends, the metrics for algorithm A and algorithm B are compared, and a choice between the two approaches is made according to those metrics.

A/B testing is not the only way to run online tests. There are more elaborate mechanisms for online testing that go beyond A/B testing, and bandit algorithms are one example. At this stage you only need to know that these approaches exist. The mechanics of how they choose between options are a topic for further study.