Model Evaluation Basics

Imagine you have built a model that predicts which items a user will want to see next on a shopping website. You have trained it, you have checked it, and it looks promising. Building the model, however, is only part of the work. Once a model exists, you need a way to judge how well it works, and you may also need a way to adjust the settings that shaped it.

This topic introduces two linked ideas. Advanced model evaluation is about judging how well a model performs. Parameter tuning is about adjusting the settings that shape the model. Evaluation and tuning work together: tuning produces different versions of a model, and evaluation is how you compare those versions. Read together, the two form a loop of adjust, judge, and adjust again.

The article then adds a third idea that ties the workflow together: pipelines, which chain steps so that a workflow runs in order and is wrapped into a single unit. We will start with pipelines, because the order of steps is something you can trace by hand before you ever touch a real model.

A pipeline chains steps so that each step receives the output of the previous one. The whole chain is encapsulated as one unit, so the workflow can be run as a single operation rather than as a set of separate manual actions. The steps are ordered, and the order matters, because each step works on whatever the previous step produced.

It is important to see what a pipeline does not do. It does not change what each step does. A step still does its own job. The pipeline only arranges the steps and runs them in sequence, passing each result forward to the next step as its input.

Now take the same two steps and run them in the opposite order: doubling first, then adding one. The steps themselves are unchanged. Only their arrangement is different. Trace the value again, one step at a time, and compare the final answer.

The key idea is that the model's predictions are checked against a test set that already exists before the model meets real users. Offline evaluation therefore asks a narrow question: how well do the model's predictions match the data it is being scored against? Going back to the shopping example, the team can check whether the items the model predicts a user will want actually match what the test set says those users wanted.

The test set is collected beforehand, which means it is fixed and known before the model is scored on it. Because the score is taken from this fixed set, different versions of a model can be compared on the same footing. This is what makes evaluation useful for the tuning loop: you tune, score each version on the same test set, and keep the version that scores better.

Once an algorithm is part of a live website, the question changes. It is no longer enough to ask whether the predictions match a test set. The team must ask what happens to the whole system when users meet the new behavior. This is called online testing, also known as live testing. It evaluates the consequences of employing the algorithm in the overall system.

Online testing usually comes next, after offline evaluation. The reason is that user behavior can change in unexpected ways. A model can score well on a test set and still shift how people use the website once they see its recommendations. Offline evaluation cannot show those shifts, so for user-facing systems the live check is not something to skip.

Online testing has several methods. The most common protection against unexpected behavior is A/B testing. Another family, bandit algorithms, goes further. Both evaluate an algorithm's consequences with live use, and the two are described below.