Designing Feedback Systems
Feedback from users is shaped by the setting in which it was given, and the layout of the page is part of that setting.
A Click That Looks Like a Vote
Imagine a search box that returns five results for a query. Over a week, the team notices that the first result gets clicked far more often than the others. The natural conclusion is that the first result is the best one. Before accepting that conclusion, it helps to ask a more careful question: would the same result have been clicked if it had appeared third?
Feedback from users is shaped by the setting in which it was given, and the layout of the page is part of that setting. Feedback systems often treat every click as a vote for the clicked item. This article shows why that reading can mislead, and it looks at two families of distortion that appear in feedback. The first depends on where an option appears. The second depends on the person giving the feedback and on the way options are presented to them.
Following One Click Through the List
To see how position can distort feedback, follow a single suggestion through the sequence of events that produces a click. The steps below trace that chain from the moment the list is displayed to the moment the feedback is recorded.
- 1The system displays a list, and each suggestion occupies a particular slot in that list.
- 2The user's attention moves through the list in an order, so the first item they encounter has a different chance of being selected than the item below it.
- 3The user clicks one suggestion, and the click depends partly on the slot that suggestion occupied.
- 4The recorded click does not carry a label saying where the option was when it was chosen. The system only sees that a click happened.
- 5A naive analysis treats that click as a vote for the option's quality, even though the slot may have contributed to it.
The highlighted top slot is the one the source describes as receiving the advantage. Users are generally more likely to click the first suggestion than the second. The diagram shows the layout, and the layout is the reason the top suggestion is seen first and has the easiest path to a click. Nothing in the recorded click reveals that the suggestion was sitting in slot 1.
Why a Top Click Is Not Proof of Quality
Two claims sit at the core of this idea. The first is that users are generally more likely to click on the first suggestion than on the second. The second is that a click on the first suggestion does not, by itself, establish that the suggestion is good.
Together these claims mean the raw click record mixes two things: the suggestion's real merit and the advantage that its position gave it. A feedback system that reads clicks at face value will reward whatever sits at the top. Over time it may keep rewarding that item even when a better one sits lower on the list, because the lower item never receives the same chance to be clicked.
Reading the search box results
A search box returns five results. The first result gets clicked far more often than the others over one week. Does that show the first result is the best?
- 1Record the observation
The first result receives many more clicks than the other four results.
- 2State the natural conclusion
The tempting reading is that the first result is the best, since it attracts the most clicks.
- 3Ask the counterfactual question
Would the same result have been clicked if it had appeared third? If the answer might be yes, the click count is partly a product of position.
- 4Separate the two causes
The raw click count mixes the result's merit with the advantage of the top slot. Until those are separated, the click count cannot settle which result is best.
The click count alone does not establish that the first result is the best. The slot may explain much of the difference.
Two Ways to Reduce Position Bias
The source identifies two mitigations. Both aim to stop position from being mistaken for quality, but they work in different ways. The first changes what the user sees. The second changes how the system interprets what the user did.
Preference Bias in Side-by-Side Comparisons
Position is not the only thing that distorts feedback. The person giving feedback brings their own tendencies, and the source describes these as preference biases. Many such biases can affect a person's judgment, and some of them show up sharply when people compare two outputs side by side. The two cases the source highlights are a preference for longer responses and a tendency to favor whichever answer was seen last.
Two Kinds of Bias Compared
Both kinds of bias distort feedback, but they come from different places. Position bias comes from the layout the user is shown, and it applies to any list of options regardless of who is looking at it. Preference bias comes from the person giving feedback and from how they compare what they see. The table sets out the contrast.
| Question | Position bias | Preference bias |
|---|---|---|
| Where does it come from? | The layout of the list the user sees | The person giving feedback and how they compare options |
| What does it favor? | Whatever sits in the top slot | Longer responses, and the answer seen last |
| Does it need a comparison? | No, it applies to any list of options | It appears most sharply in side-by-side comparisons |
| How is it reduced? | Randomizing positions, or modeling success rate by position | Inspecting feedback for these tendencies before interpreting it |
Position bias and preference bias compared
Inspecting Feedback Before Product Decisions
The source makes a practical point that ties the two biases together: it is important to inspect user feedback to uncover its biases. Understanding them helps you interpret the feedback correctly, which in turn helps you avoid misleading product decisions. The sequence below is one way to put that inspection into practice. It is a suggested workflow built on the biases described above, not a fixed procedure from the source.
- 1Collect the raw feedback, including which slot each option occupied and the order in which two responses were shown.
- 2Check for position bias by grouping clicks by slot and asking whether the slot alone accounts for much of the difference.
- 3Check whether the feedback was gathered with randomized positions, or whether a fixed order could be favoring one option.
- 4Check side-by-side comparisons for a lean toward longer responses and for an advantage given to the answer seen last.
- 5Interpret the feedback in light of these checks, and only then turn it into a product decision.
Mistakes When Reading Feedback
Common mistake Treating the most-clicked item as the best item
What it looks likeThe first result gets the most clicks, so the team ships it as the top ranking.
Why it failsUsers are generally more likely to click the first suggestion, so the click count mixes position with merit.
Do this insteadCheck how the same item performs in other slots, or model success rate after accounting for position.
Common mistake Assuming randomization is unnecessary when clicks already look sensible
Why it failsPosition effects can be present even when the ranking looks plausible, and they are invisible in the raw click record.
Do this insteadRandomize positions, or model success rate by position, before trusting the comparison.
Common mistake Reading a preference for a response as a judgment of quality without checking length
What it looks likeThe longer of two responses wins most comparisons, so the team concludes that the longer responses are better.
Why it failsPreference bias includes a tendency to favor longer responses even when they are less accurate.
Do this insteadInspect the comparisons for a length effect before drawing conclusions about accuracy.
Common mistake Ignoring the order in which side-by-side options are shown
Why it failsRecency bias favors the answer seen last, so a fixed presentation order can build that bias into the recorded preference.
Do this insteadVary or account for presentation order when interpreting side-by-side results.
Checking a Ranking Before You Trust It
A list of five results gets most of its clicks in slot 1. Before deciding that the slot 1 result is the best, what two things would you check in the feedback, and why does each matter?
Hints
- Think about what the recorded click does and does not say about the slot where it happened.
- Consider whether the same result was ever shown in a different slot, and how randomization or a position-aware model would help separate the two effects.
Key Takeaways
- Position bias means the slot an option occupies changes how it is perceived. Users are generally more likely to click the first suggestion, so a click at the top does not prove that the suggestion is good.
- Two mitigations reduce position bias: randomly varying the positions of suggestions so no option is systematically favored, and modeling each suggestion's success rate while accounting for the slot where it was shown.
- Preference bias comes from the person giving feedback. It includes a tendency to favor longer responses even when they are less accurate, and recency bias, which favors the answer seen last in a side-by-side comparison.
- Position bias comes from the layout the user sees, while preference bias comes from the reviewer and how they compare options. Both must be checked before feedback is interpreted.
- Inspecting user feedback for these biases is necessary to interpret it correctly and avoid misleading product decisions.