Understanding Offline and Online Evaluation

Suppose you have built a recommendation model and it performs well on a test set that you collected before deployment. That result is encouraging, but it answers only part of the question. A model that scores well on past data may still change what real people do once it is placed in front of them. This article looks at the next step in evaluation, the one that happens after offline checks are done, and at one of the most common tools used to carry it out.

Keep one question in mind as you read: when an algorithm changes what a user sees, what evidence would tell you whether that change actually helped the people using the system?

Offline evaluation is a necessary starting point. It is cheap to repeat, it does not expose real users to an unfinished system, and it lets you compare candidate models on identical inputs. For a user-facing machine learning system, however, it is only the first step in evaluating an algorithm. The predictions it measures are fixed in advance, while the system's behavior in production depends on how people respond to what they are shown.

The reason this step matters is that changing the recommendations or search results a website shows can drastically change user behavior. A change that looks like a clear improvement in a test set can lead to unexpected consequences once people start reacting to it. Because of this, online testing is the usual next step after offline evaluation for systems that users see directly.

A/B testing is a common form of online testing. The procedure has three stages. First, users are divided into two groups and each group receives one algorithm. Second, relevant success metrics are recorded for both groups over a set period of time. Third, the metrics of algorithm A and algorithm B are compared, and a choice between the two approaches is made according to those metrics.

Follow one incoming user through this process. The user arrives, is placed into one of the two groups, and receives the algorithm assigned to that group. The diagram below shows the path every user takes. Notice that the two branches are the only place where the experience differs, and that both branches feed the same measurement and comparison steps.

The users never need to know that a comparison is happening, which is why the study is described as blind. From the user's side, the experience looks normal. From the evaluator's side, the two groups are the controlled difference that the comparison depends on.

A common arrangement is for algorithm A to be a new model and algorithm B to be the established system that is already running. The new model is tested against the current one on live traffic, so that the decision to adopt it rests on how users actually behave rather than on offline scores alone.

The main benefit of A/B testing is that it lets you evaluate algorithms in the wild. Because the users are interacting with the model in its real setting, the test can surface effects that no offline measurement would have predicted. This is the protection that most user-facing services rely on: a small portion of traffic absorbs the risk of a surprising change before it reaches everyone.

An algorithm is not evaluated in isolation. It is one stage in a path that runs from building a model, through offline checks, into live traffic, and finally into a decision. Online testing sits at the point where the algorithm meets real users, so the measurements it produces reflect the whole user-facing system rather than only the model's predictions.

The reason online testing is the usual next step is that the consequences of a change can travel through user behavior. A new ranking or recommendation changes what people click, and that change in behavior is exactly what the offline test set could not contain, because the test set was fixed before anyone reacted to the new output.

Once the metrics have been recorded, the comparison can be as simple as computing a rate for each group. A common example metric is the fraction of visitors who clicked on a recommended item. Each group has its own count of visitors and clicks, and the click rate for each group is computed so the two can be placed side by side.

The script below uses illustrative counts (1000 visitors per group) chosen so that the resulting rates match the values in the source example. The counts are invented for teaching purposes and are not real experimental data.

Here group A shows a click rate of 0.120 and group B shows 0.110. Those two numbers are the raw material for the comparison step. Reading them correctly requires care: a difference in rates is only the starting point, and the test is designed around the metrics that were chosen in advance rather than whichever numbers happen to look best.

The procedure is simple to describe, but several errors come up repeatedly when people read or run these tests. The points below are drawn directly from how the test is meant to work.

A/B testing is not the only way to run online tests. There are more elaborate mechanisms for online testing that go beyond A/B testing, and bandit algorithms are one example. A/B testing is a well-established starting approach within a larger set of options.

You do not need to master bandit algorithms to understand the core idea of live evaluation, but it is useful to know they exist when a simple two-group comparison is not the right fit.