Online Testing in Production

Imagine you have built a model that scores well on a test set you collected in advance. It is tempting to conclude that the model is ready to serve real users. For a system that users interact with directly, such as a recommendation feed or a search engine, that conclusion skips an important step.

The reason is that the model's predictions are only one part of a larger system, and the people using that system respond to what they are shown. A list of recommended items or a set of search results is not a neutral output. Once people see it, they may click differently, stay for different lengths of time, or move through the site in new ways. A strong score on a fixed dataset cannot tell you how those responses will play out.

This article follows the path machine learning teams take from checking predictions on a fixed dataset to measuring what happens when an algorithm is live. You will see why A/B testing is the standard tool for that transition, what kinds of surprises it can uncover, and where bandit algorithms fit in as a more elaborate extension.

Notice what is absent from that process. Nothing is deployed to users. The model never reaches a real person, so the only thing being measured is how well its predictions match answers that were already known. This makes offline evaluation the natural starting point: it is controlled, and the same test set can be reused again and again to compare candidate models under identical conditions.

For a user-facing system, offline evaluation is only the first stage. The next stage is usually online testing, sometimes called live testing. Online testing evaluates the consequences of employing an algorithm in the overall system, so the question shifts from how accurate the predictions are to what the algorithm does once it is running with real users, real traffic, and real behavior.

Read the diagram from left to right. The first two stages happen with no users involved. Only after the model passes that check does it move into live testing, where the thing being measured is the outcome for the system as a whole.

The central reason online testing matters is that changing what users are shown can change how they behave, and that behavior change is part of the system's outcome. Offline evaluation cannot observe these effects because the users who would react are not part of the test set at all.

In an A/B test, a selected portion of users is served the service or website running algorithm A, while the remaining users are served the service running algorithm B. For both groups, relevant success metrics are recorded over a set period of time. When that period ends, the metrics for the two algorithms are compared, and a choice between the two approaches is made according to those metrics.

Follow one user through the diagram. The user arrives, the traffic split assigns them to a group, and the algorithm serving that group determines what they see. Their resulting behavior is recorded as part of their group's success metrics. The two groups are then compared side by side. Each user contributes to only one group, so the comparison is between the two algorithms, not between two versions of the same user.

In practice, algorithm A is often the new model being evaluated, and algorithm B is the established system that is already running. The established system serves as the baseline, and the question is whether the new model does better on the metrics that matter. Keeping the baseline running for the other group gives the team a live reference point, rather than comparing the new model against an imagined expectation.

The main benefit of A/B testing is that it lets you evaluate algorithms in the wild, meaning under the actual conditions in which users interact with the product. This is where unexpected consequences tend to appear. A change that looked harmless on a test set may alter what users see, and therefore alter what they do. Because the change reaches real users, its side effects become visible in the recorded metrics.

The common thread is that a fixed test set freezes the world at one moment and only includes the predictions, not the people reacting to them. Live testing restores the reacting people, which is where the surprises come from. The source describes these effects as the consequences of employing an algorithm in the overall live system.

A/B testing is not the only way to run online tests. There are more elaborate mechanisms that go beyond it, and bandit algorithms are one example. They belong to the same family of online testing methods, and they build on the same basic idea: testing algorithms in a live system rather than only on a fixed dataset.

At this stage of your learning, the key point is awareness. Bandit algorithms are a natural next topic once the fixed two-group design of A/B testing is clear. Recognizing that they are a more elaborate family of online testing methods is enough for now; their detailed mechanics are a subject for later study.