A/B Testing for AI Models

Suppose a team wants to find out which of several AI models gives answers that people prefer. Each time they collect feedback, a person is looking at a response to a prompt. The question to think about is how many candidate responses that person is shown at once. Your answer to that question determines which method you are using, and the two possible methods are easy to confuse even though they produce very different kinds of data.

A common first impression of comparative evaluation is that it means showing a user one model's answer, then another model's answer, and recording which one the user liked more. That description sounds reasonable, but it describes a different method. Comparative evaluation has a specific structure, and mixing it up with a neighboring technique leads to wrong conclusions about what the results mean.

In A/B testing, one candidate is shown at a time. Each person who sees a request gets a single candidate response, and the feedback describes that one candidate. Comparative evaluation works differently. Because the evaluator sees several candidates together, each judgment is a direct contest between models. The evaluator is not scoring a response against an absolute standard; the evaluator is choosing between responses that were produced for the same request.

This difference is why the two methods are not interchangeable. An A/B test tells you how people respond to one candidate. A comparative evaluation tells you which of several candidates won when they were placed head to head. The raw data also has different shapes: an A/B test produces feedback about individual candidates, while comparative evaluation produces a history of head-to-head results that can be ranked.

Comparative evaluation records each contest as a row: which two models competed, and which one won. Each row is one judgment by an evaluator, and no single row says anything about a model in isolation. The table below is a small history of three matches, written here as an illustrative example.

This raw history does not yet answer the question a team usually cares about. Three matches are too few to say whether Model A is better than Model B in general. To answer that kind of question, the matches between the same pair of models have to be summarized, and the standard summary is the win rate.

The win rate of model A over model B is the probability that A is preferred over B. It is estimated by looking at every match between A and B and calculating the percentage of those matches that A won. The calculation is a simple count: wins divided by total matches, expressed as a percentage.

Source data often arrives already summarized as win rates for each pair of models, rather than as raw match rows. In that form, each value still means the same thing: the percentage of matches in that pair which the first model won. Source data of this kind can describe many pairs at once, with the matches behind each value often counted in the same way across pairs.

With two models, ranking is straightforward: the model that wins more often ranks higher. With five models, it is no longer obvious how to order them from a table of ten pairwise win rates, because some models win against some opponents and lose to others. This is the job of a rating algorithm. Typically it first assigns each model a score based on the comparative signals, then ranks the models by those scores.

Several rating algorithms have been adapted from other fields, including Elo, Bradley-Terry, and TrueSkill. They come from sports and video games, where ranking players from head-to-head results has been done for a long time. The choice of algorithm matters because different ranking algorithms can produce different rankings from the same comparison data. A ranking can be judged by how well it predicts future match outcomes.

Comparative evaluation measures which response an evaluator preferred. That is a useful measurement when preference is what you care about, but it does not automatically measure the quality you care about for every question. A preference judgment records what the evaluator chose between the two candidates shown. It does not, by itself, confirm that the chosen answer is factually correct.

Consider a factual question where one response is fluent and confident but wrong, and the other is correct but plainly worded. An evaluator who prefers the fluent answer produces a win for the wrong model. The win rate is then accurate as a record of preference, yet it misleads anyone who reads it as a measure of correctness. Preference-based comparison is therefore a weak signal for questions where the right answer can be checked and where style can pull the evaluator away from accuracy.

Comparative evaluation places two or more models in head-to-head matches on the same request, and an evaluator picks the winner of each match. That structure is different from A/B testing, where one candidate is shown at a time. The recorded matches can be summarized as win rates, and a rating algorithm can turn those comparisons into a ranking of several models. Each step has a limit: win rates need enough matches, rankings depend on the algorithm, and preference is not the same as correctness.