Model Evaluation Fundamentals
A rise in purchases is a single total produced by several candidate causes. Evaluating a model means isolating its contribution, not just noticing that the total went up.
A Sales Rise With Several Possible Causes
Imagine a store switches on a new recommendation model, and within a month its purchases climb. It is tempting to credit the model right away. Yet the same month may also have brought a discount campaign or the launch of a new product line. Purchases can rise for many reasons at once, and the sales total by itself does not reveal which reason mattered. This article follows that question step by step: how to tell whether the model deserves the credit, and how an A/B test is designed to answer it.
What do you think happens?
The store turned on a recommendation model and purchases rose the same month. Can the rise be credited to the model based on that total alone?
Reveal answer
No, the total alone cannot show what caused the rise
A promotional campaign or a new product launch in the same period could also increase purchases. The overall sales total does not separate these causes, so it cannot show how much the model contributed.
Tracing the Causes Behind a Purchase Rise
When we ask why purchases changed, we are really asking about a list of possible causes. Recommendations are one entry on that list, and it is the one we usually care about when we evaluate a model. Two other entries are easy to overlook. Promotional campaigns can push people to buy. New product launches can attract buyers who would not have shopped otherwise.
The difficulty is that these causes often overlap. If the model, a campaign, and a launch all start in the same period, they all change together. A simple before-and-after comparison then cannot tell us how much each one contributed. The timeline below traces how three changes landing in one month produce a single rise that could have come from any of them.
A rise in purchases is a single total produced by several candidate causes. Evaluating a model means isolating its contribution, not just noticing that the total went up.
Splitting Shoppers Into Two Groups
A/B testing compares two versions of a model or system to determine which one performs better, and it is the tool used to separate the impact of recommendations from other factors. The central idea is to run two versions at the same time. One version uses the recommendation model, and the other does not. The version without the model is the control, and the version with it is the treatment.
Shoppers are divided into the two groups before the outcome is measured. This ordering is what makes the comparison fair. Because both versions face the same promotions and the same product launches during the test, those outside factors affect both sides roughly equally. The comparison below shows what each group shares and what differs between them.
Reading the Gap Between Two Groups
A store splits incoming shoppers evenly into a control group and a treatment group for one month. Both groups see the same discount campaign and the same new product line. Assume (illustratively) that the control group made 300 purchases and the treatment group made 360 purchases. What does the gap show about the model?
- 1Identify the shared conditions
The campaign and the product launch affected both groups during the test, so they cannot explain the difference between the groups.
- 2Identify the single difference
The only deliberate difference between the groups is whether the recommendation model was active.
- 3Compute the gap
The gap is 360 minus 300, which is 60 additional purchases in the treatment group.
- 4Read the gap, not the total
The model's contribution is judged from this gap. The combined total of 660 purchases would not reveal it.
The 60-purchase gap is the difference the test attributes to the recommendation model, because the promotion and launch were shared by both groups.
Mistakes When Reading A/B Results
The same sales data can be read in two ways. The first looks only at the overall change before and after the model was turned on. The second uses the two groups that the A/B test created. The table below contrasts them.
| Question | Before-and-after reading | Concurrent control-versus-treatment reading |
|---|---|---|
| What is compared? | Purchases in one period versus purchases in another | Purchases of two groups running at the same time |
| Are outside factors shared? | Not reliably, since promotions and launches may differ between periods | Yes, both groups face the same promotions and launches |
| What is the signal? | The overall change, which mixes all causes | The gap between the groups, which isolates the version |
| Can it credit the model? | No, the change may come from other causes | Yes, more plausibly, because the version is the main difference |
Two ways of reading the same sales data
Common mistake Comparing purchases before and after the model was turned on
What it looks likePurchases were 500 last month and 620 this month, so the model must have added 120 purchases.
Why it failsThe later month may also contain a discount campaign or a new product launch. The before-and-after change mixes all of these causes together.
Do this insteadRun the two versions at the same time, with one group using the model and one not, and compare the groups.
Common mistake Reading the overall total instead of the gap between groups
What it looks likeTotal purchases across both groups rose, so the model is working.
Why it failsThe total includes the effect of shared conditions that both groups experienced, so it does not show the model's contribution.
Do this insteadRead the difference between the control and treatment groups.
Common mistake Forming the groups after seeing the outcome
What it looks likeChoosing the shoppers who already bought more and placing them in the treatment group.
Why it failsThe source stresses that shoppers are divided into groups before the outcome is measured. Choosing groups after the fact removes the fairness of the comparison.
Do this insteadDivide shoppers into the two groups before measuring purchases.
A before-and-after comparison is the most natural mistake because it looks like a clean experiment. It does not hold the promotions and product launches fixed, so the model cannot be separated from them.
Checking Your Reading of a Test
A store runs an A/B test for two weeks. Both groups see the same holiday promotion and the same new product line. The treatment group, which receives recommendations, makes 25 more purchases than the control group. A colleague says the test proves the model caused the whole rise in purchases during those two weeks. Explain what the 25-purchase gap does and does not show, and name the mistake in the colleague's claim.
Hints
- Ask which factors were shared by both groups and which factor was different.
- Consider whether the overall rise or the gap between groups is the right measure of the model's contribution.
- Think about what the promotion and launch would do to the control group if they were removed from the comparison.
Key Takeaways
- A rise in purchases alone does not show that a recommendation model caused it, because promotional campaigns and new product launches can also increase purchases.
- A/B testing runs two versions at the same time, one with the recommendation model and one without, so shared outside factors affect both sides roughly equally.
- Judge the model by the gap between the groups, which is divided before the outcome is measured, rather than by the overall total or a before-and-after change.
- Common mistakes include comparing before and after periods, reading the overall total instead of the gap, and forming groups after seeing the outcome.