Offline Evaluation of Machine Learning Models

Imagine a recommendation model that scores very well on a carefully prepared set of examples. Its predictions look accurate, its numbers look strong, and it seems ready to ship. The question is what happens next. Once real people start seeing its recommendations on a live website, their behavior may change in ways the test set never captured. This article explains how teams evaluate an algorithm once it is part of a working product, where people react to what it shows them.

Offline evaluation is where most evaluation starts. It judges an algorithm on a test set collected beforehand. Nothing about this step involves real users reacting to the algorithm. The test set is fixed, so results can be reproduced and candidate models can be compared with one another without putting anything in front of customers.

For a user-facing machine learning system, however, offline evaluation is only the first step. It tells you how the algorithm performs on past data. It does not tell you what happens when the algorithm's output is shown to people and they respond to it. The live test comes after the offline stage, as the next step in the same progression.

The reason offline evaluation is not enough is that changing what users see can change how they behave. Altering the recommendations or search results a website shows can have a large effect on user activity, and that effect may be hard to predict from offline numbers. Online testing, also called live testing, exists to evaluate these consequences inside the real-world system.

Online testing is built to surface unexpected consequences before they affect everyone. The source describes these as effects that appear only when real people interact with the system. Offline numbers can miss them because the stored test set never contains the feedback loop that a live product creates.

Most user-facing services use A/B testing to protect against surprises. In an A/B test, a selected portion of users is served the website or service running algorithm A, while the remaining users are served the version running algorithm B. Neither group knows which one it belongs to. Tracing a single arriving user through this process shows how the split works.

A common arrangement is for algorithm A to be a new model and algorithm B to be the established system. The established system acts as the baseline. The new model has to show better metrics in live use before it is selected. This is the practical payoff of testing in the wild: you learn how real users respond, rather than relying only on the stored test set.

A/B testing is not the only way to test online. There are more elaborate mechanisms for online testing that go beyond A/B testing, and bandit algorithms are one such mechanism. The source material treats them as a more advanced topic and recommends a dedicated introduction for readers who want the details. This article only places them within the progression.