Offline Evaluation of Predictive Models

Suppose you have built a model that makes predictions, and you want to know whether it is good enough to use. A natural first move is to collect a set of examples ahead of time, hold them back as a test set, and measure how accurately the algorithm predicts on that set. This is what offline evaluation means. It is a useful starting point, but it answers a narrow question: how well does the algorithm perform on data we already have?

The score answers the question of how well the algorithm performs on data we already have. It does not answer what happens after the algorithm is deployed and starts changing what people see. That gap between stored data and live behavior is the reason offline evaluation is only the first step for a user-facing system.

Offline evaluation is the first step in evaluating an algorithm for a user-facing system. It is not the last one. Once an algorithm will change what real users see, the offline results must be followed by evaluation under live conditions. The diagram below shows the order: offline evaluation comes first, online testing follows, and bandit algorithms extend online testing.

Online testing evaluates the consequences of an algorithm inside the overall live system. The reason is that changes to what a website shows can alter user behavior in ways that no stored test set would reveal. Changing the recommendations or search results a site presents can drastically change how people behave, and that change can lead to unexpected consequences.

Follow one request through the system to see where this happens. A user makes a request, the model produces a prediction such as a recommendation, the user reacts to what is shown, and the reaction is recorded as a success metric. The model's offline score never sees that final reaction, so the consequences can only be observed by measuring the live flow.

An A/B test splits users into groups served different algorithms, records success metrics over a set period, and compares the results in the wild. In many real deployments, algorithm A is a new model and algorithm B is the established system it may replace. The steps follow a fixed order, shown in the diagram below.

The source describes the steps of an A/B test but does not specify how users should be assigned to groups in practice. The following trace is an illustration written for this lesson. It uses a simple invented rule that alternates assignments, and the metric is whether the user clicked (1 for yes, 0 for no). The point is the flow: assign each user to one group, record a metric for each user, then compare the group summaries.

The comparison in the trace above only looks at averages. A real test would also need to consider how long the test ran and whether the difference is large enough to trust. The source material does not go into those details, so this article does not extend the example beyond the comparison itself.

A/B testing is not the only way to run online tests. There are more elaborate mechanisms for online testing that go beyond A/B testing, and bandit algorithms are one such mechanism. They belong to the same family of live evaluation methods, but they are presented as more advanced than a fixed two-group comparison.

This article only introduces bandit algorithms. It does not explain how they decide which users see which algorithm, so questions such as how traffic shares shift as results accumulate are outside its scope. What you can take away is the direction of the progression: offline evaluation, then A/B testing, then bandit algorithms that extend online testing.