Online Testing and Live Evaluation
The first stage is offline evaluation, which asks whether the model's predictions are good on data the model has not seen.
Why a Good Test Score Is Not the Finish Line
Imagine you have built a model that predicts which items a user will want to see next on a shopping website. You have trained it, you have checked it, and it looks promising. The question this article answers is what kind of checking comes first, what kind comes next, and why the second kind cannot be skipped when real people are involved.
The short answer is that checking happens in two stages. The first stage is offline evaluation, which asks whether the model's predictions are good on data the model has not seen. The second stage is online testing, also called live testing, which asks what happens to the whole system once the new behavior reaches real users. The rest of this article follows the path from the first stage toward testing with live users.
What Offline Evaluation Measures
Offline evaluation assesses algorithmic predictions on a test set collected beforehand. It is the first step for user-facing systems.
The key word is beforehand. The test set is gathered before the algorithm is placed in front of anyone. Because the data already exists, the team can run the algorithm over it and compare its predictions with the known outcomes. Nothing about the live website has to change, and no user has to experience the new algorithm.
What offline evaluation measures is the quality of the predictions themselves. It answers the question of whether the predictions match the test set. It does not, by itself, answer what the algorithm will do to the behavior of people using the site.
Why Online Testing Comes Next
Once an algorithm is part of a live website, the question changes. It is no longer enough to ask whether the predictions match a test set. The team must ask what happens to the whole system when users encounter the new behavior. This is the purpose of online testing: the consequences of employing the algorithm in the overall system are evaluated.
Online testing usually comes next because user behavior can change in unexpected ways. A prediction that scores well on historical data can still lead people to act differently once they see it, and those changes are only visible when real users are involved. This is why the second kind of checking cannot be skipped for systems that people use.
Offline Evaluation Versus Online Testing
The two kinds of checking look at different things. Offline evaluation looks at the predictions against a test set. Online testing looks at the consequences of the algorithm inside the overall system, with real users encountering the new behavior. Each one tells the team something the other cannot.
Offline evaluation can tell a team that the predictions are accurate on the test set. It cannot reveal how users will change their behavior once they see those predictions on a live site. Online testing is the only one of the two that observes those changes directly.
How A/B Testing Splits Traffic
The most common protection against these surprises is A/B testing, which is a form of blind user study. In A/B testing, a selected portion of users is served the website or service using algorithm A, while the remaining users are served algorithm B. The mechanism is easiest to follow by tracing one incoming user through the system and then tracing the data that is recorded.
Tracing an A/B Test Step by Step
Tracing an illustrative A/B test
A team has a new recommendation model (algorithm A) and an established system (algorithm B). They want to know which one leads to better results for users. The numbers below are invented for illustration only; the source does not give any figures.
- 1Select a portion of users
A selected portion of incoming users is served algorithm A. For this illustration, assume 20 percent of visitors are placed in the group that receives A.
- 2Serve the remaining users
The remaining users are served algorithm B, the established system, which keeps serving everyone else as before.
- 3Record success metrics for both groups
Over a set period, the team records a relevant success metric, such as the share of visitors who click a recommended item. Suppose group A shows 5 percent and group B shows 4 percent.
- 4Compare and choose
The metrics for A and B are compared. The choice between the two approaches is made according to those metrics, so the team has evidence about the live behavior that the test set could not supply.
The new model is preferred on the chosen metric only because the live comparison favored it. The offline test set alone would not have shown how users in each group responded.
Predicting Who Knows About the Test
What do you think happens?
In an A/B test, do the users receiving algorithm A know that they are in a test group?
Reveal answer
No, they do not know which group they belong to.
A/B testing is described as a blind user study. Users do not know which group they belong to, so their responses reflect ordinary use rather than a response to being tested.
Bandit Algorithms as a Broader Family
A/B testing is not the only method for online testing. There are more elaborate mechanisms that go beyond A/B testing, and bandit algorithms are one example. Bandit algorithms belong to the online testing family, so they also evaluate an algorithm's consequences through live use. They are more involved than a simple split between two fixed groups.
Mistakes When Reading Online Results
Common mistake Treating a strong offline test score as proof that the live system will improve
What it looks likeA model matches the test set well, so the team skips live testing and ships it to every user.
Why it failsOffline evaluation measures predictions against a test set collected beforehand. It does not show how user behavior changes once the new behavior reaches real people.
Do this insteadUse offline evaluation as the first check, then run online testing to observe the consequences in the overall system.
Common mistake Assuming users in an A/B test know which algorithm they are seeing
What it looks likeA team expects users in group A to behave differently because they realize they are being tested.
Why it failsA/B testing is a blind user study. Users do not know which group they belong to.
Do this insteadKeep the groups blind, and compare the recorded success metrics for each group.
Common mistake Reading the comparison without recording metrics over a set period
Why it failsThe source describes recording relevant success metrics over a set period of time before comparing A and B.
Do this insteadDecide the success metrics and the time period in advance, then compare the recorded values for both groups.
Your Turn to Choose a Checking Stage
A team has a new model that scores well on a test set collected beforehand. Explain in two or three sentences why the team should still run an online test before treating the model as a success for a live website. Name the stage each check belongs to.
Hints
- Think about what the test set can and cannot show about user behavior.
- The source says online testing usually follows offline evaluation for user-facing systems.
Key Takeaways
- Offline evaluation assesses an algorithm's predictions on a test set collected beforehand, and it is the first step for user-facing systems.
- Online testing, also called live testing, evaluates the consequences of the algorithm inside the overall system and usually follows offline evaluation because user behavior can change in unexpected ways.
- A/B testing serves a selected portion of users algorithm A and the remaining users algorithm B, without their knowledge, then records success metrics for both groups and compares them to make a choice.
- Bandit algorithms are a more elaborate family of online testing methods that go beyond A/B testing, though the source does not describe their internal workings.