Offline Evaluation

Building a machine learning model is only half the job. Before anyone relies on its predictions, you need evidence that those predictions are good. The first and most common source of that evidence is offline evaluation: checking algorithmic predictions against a test set that was collected beforehand. The model never has to face live users during this stage. You hold back some data, let the model predict on it, and compare those predictions with the answers you already know.

This article follows that question step by step. First you will trace how offline evaluation scores predictions item by item. Then you will see why a strong offline score is only the first step for a user-facing system, and how online testing, including A/B testing, examines what happens once real people interact with the algorithm.

Offline evaluation starts with a test set. You collect a set of examples ahead of time, and for each one you already know what actually happened. These known answers are the true labels. You then let the algorithm make a prediction for each of the same items. Scoring means lining up each prediction with its matching true label and counting how often the two agree.

Because the test set was gathered before the evaluation begins, the scoring is repeatable. You can rerun the same comparison on the same data and check whether a change to the model really helped. This repeatability is the main strength of offline evaluation, and it is a large part of why the method is so widely used early in development.

The example below makes this concrete. It uses invented labels and invented predictions for eight test items. The true labels are the answers collected beforehand, and the predictions are what the model produced for the same items. The score here is simply the number of matches divided by the number of items.

Trace the output by hand against the diagram. Items 1, 2, 4, 5, 7, and 8 match. Items 3 and 6 do not, because item 3 was predicted 0 when the truth was 1, and item 6 was predicted 1 when the truth was 0. Six matches out of eight items gives the score of 0.75 printed at the end.

A good offline score tells you that the model predicts well on the data you held back. It does not tell you what happens when the model's output is shown to people. In a user-facing system, the predictions become part of the product. They change what users see, and users then respond to what they see.

The source makes a specific point here. Changing the recommendations or search results a website shows its users can drastically change how those users behave. A recommendation list that performs well in an offline test may shift what people click, what they buy, or how long they stay, and those shifts can ripple through the rest of the system in ways no one predicted. Offline evaluation cannot reveal these effects, because the test set does not respond to the system.

The two stages answer different questions. Offline evaluation asks how well an algorithm's predictions match a test set collected beforehand. Online testing, also called live testing, asks what happens when that algorithm is actually part of the product and real users interact with it. Online testing evaluates the consequences of employing an algorithm in the overall system, not just the accuracy of its predictions.

Most user-facing services use A/B testing to observe the consequences of an algorithm. A/B testing is a form of blind user study. Without their knowledge, a selected portion of users is given a website or service that uses algorithm A, while the rest of the users are given the same kind of service using algorithm B. Because users do not know which version they see, their behavior reflects ordinary use rather than how they act when they know they are being tested.

Trace one user through this flow. The user arrives and is placed into one group without noticing. If that user lands in the algorithm A group, every success metric recorded for that person counts toward group A. The same user never contributes to group B, which is why the two groups must be kept separate. Both groups are observed for the same set period, and when that period ends the metrics for A and B are compared.

The source notes that often A is a new model while B is the established system. The test therefore asks whether the new model earns its place in front of real users. A choice between the two approaches is made according to the recorded metrics.

A/B testing is not the only way to run online tests. The source mentions more elaborate mechanisms for online testing that go beyond A/B testing, and it names bandit algorithms as one example. At this stage the important point is that bandit algorithms belong to the online testing family. Like A/B testing, they evaluate algorithms through live user interaction, but they go beyond a simple fixed split between two groups.

The mechanics of how bandit algorithms choose between options are a topic for further study. For now, it is enough to know that these approaches exist and that they, too, are a form of online testing rather than offline evaluation.