Biases in User Feedback

Most people picture feedback as a deliberate act: a star rating, a thumbs-up button, or a text box asking for comments. Yet a product can learn a great deal without asking the user anything at all. Every choice a person makes while using a tool carries information about whether the tool helped. Which of several results did they pick? Did they accept the suggestion? Did they keep typing past it? These actions already belong to the user's workflow, and the signals they produce are called implicit feedback.

This article follows one idea through several products. Implicit feedback is useful, but every signal it produces has to be interpreted, and that interpretation is where bias enters. By the end you should be able to trace an action from the moment a user performs it to the moment a team records it, and to spot the places where the record can drift away from what actually happened.

The two kinds of feedback differ in where they come from. Explicit feedback is produced on purpose: the user notices a reason to respond and spends effort on the response. Implicit feedback is a by-product of work the user was already doing. The user did not set out to grade the product; they were trying to get something done, and the grade is inferred afterward from what they did.

Understanding this distinction helps you decide what to measure, how to present choices, and how much trust to place in the signals you receive. A team that treats an inferred signal as if it were a stated opinion will often overtrust it.

Consider a code assistant that proposes a completion while a developer types. The suggestion does not have a single outcome; it moves through states. It is shown. Then the developer might accept it, edit it, reject it, or ignore it by continuing to type past it. Each of these states is a possible source of a signal, but the team only sees what its product actually records.

Read the diagram as a set of possible paths, not a guaranteed sequence. Only one path is produced per suggestion, and different products record different parts of it. A suggestion that is accepted is a fairly clear positive signal. A suggestion that is ignored is much harder to read, because the developer may have kept typing for many reasons unrelated to the suggestion's quality.

Image generators often show a user several candidate images and record which one is chosen. That choice is implicit feedback. The team may read it as a preference for the image's content, but the choice also depends on where the image appeared on the screen. Imagine four candidates that are equally good, laid out in a row. The position of each option is a fact about the layout, not about the image, yet it is part of what the user sees when deciding.

Notice what this does to interpretation. If the team sees that the first position is chosen more often, the record alone cannot tell them whether the image or the placement caused the difference. The signal is real, but its meaning is ambiguous. The next section explains why one logged action can mean so many different things.

A key source of bias is that an implicit signal rarely has one meaning. Not selecting an image, for example, could mean the user disliked every option, preferred a different option, did not look closely, or could not judge the differences at all. Not accepting a code suggestion could mean it was wrong, that it was right but not what the developer wanted at that moment, or that the developer was simply not finished typing. The logged action is identical in each case, so the record does not distinguish them.

The source makes a related point about controls. A feedback control's meaning depends on how users interpret it. A rating scale is not a neutral instrument. Imagine a form where five stars is the positive end but the one-star slot carries an angry face while the five-star slot carries a neutral one. Users may read the scale in a way the designer did not intend, so the ratings can contradict the written responses that accompany them.

Several design decisions shape what users are willing and able to tell you, and visibility is one of the strongest. Users tend to act more candidly when their activity is private, because they expect less judgment from others. Private signals can therefore be of higher quality. As an example from the source, in 2024 X (formerly Twitter) made likes private, and its owner reported a significant rise in likes afterward.

Private signals carry costs, though. They reduce discoverability, because users cannot find what their connections liked. They also reduce explainability: a recommendation based on hidden likes can confuse the user about why it appeared. A team that gains cleaner data by hiding activity may lose the ability to explain the product's behavior to the people using it.

Context is not free to collect. Gathering the surrounding conversation may expose personally identifiable information, which means it often requires explicit user consent. An application built into a user's daily tools can observe what happens to its output afterward, but a standalone chat assistant cannot. Context is therefore frequently the piece that goes missing between the user's action and the stored feedback.

One practical answer is a user data donation flow. Some products include terms in their service agreements that let them access user data for analytics and product improvement. Products without such terms can instead ask users to donate recent interaction data when they submit feedback, for example through a checkbox that shares the last few exchanges as context. The diagram traces how that donated context moves from the user to the team.

The raw click records that the answer was judged negative. The donated context records the conversation that produced it, which is what makes a deeper diagnosis possible. The donation step is valuable only when the user has agreed to it; without consent, the record is limited to the click itself.

Another consideration is what you ask users to do. Comparative questions, where a user chooses between two outputs, only produce useful data when the user can actually judge the difference. When a question has a correct answer, such as a mathematical result, asking people to prefer one answer over another misrepresents the task. The comparison invites a preference where none is meaningful.