Collecting and Using User Feedback

Imagine you have two versions of an AI application and one clean number to compare them with. The habit most people reach for is to pick the version with the higher overall score and call it the winner. A single number feels fair, easy to explain, and easy to defend. This article examines when that habit leads you wrong, and how user feedback and careful evaluation can give you a more accurate picture before you ship.

The belief behind the single-score habit is that the overall number tells you how the system performs for everyone, so a higher number means a better system across the board. That belief breaks down when the overall number is computed over groups of different sizes. An overall percentage is not a neutral average. It mostly reflects where each model's examples happened to fall.

Consider what that means in practice. A model can be better in every individual group and still lose the overall comparison, if the model with the lower per-group numbers has most of its examples in the group where performance is highest. The overall figure then rewards that model for the composition of its test cases rather than for its behavior.

Work through the evidence for the two models. Both were tested on a set of 350 examples, split into Group 1 and Group 2. Model A has 87 examples in Group 1 and 263 in Group 2. Model B has 270 examples in Group 1 and 80 in Group 2. The group sizes are very different between the two models, and that difference is what drives the overall result.

Read the diagram column by column. Model A's examples are concentrated in Group 2, and Model B's are concentrated in Group 1. Because the overall figure weights each model's examples by where they fall, the two overall numbers are answering a question about example mix, not about which model handles each group better. Model A is higher in both groups, which means Model A is the stronger model in the sense that matters for each kind of case, even though Model B's overall number is higher.

The defense against Simpson's paradox is slicing. Slicing means separating your data into subsets and looking at system performance on each subset separately. Slicing is not a special technique reserved for paradoxes. It is a general habit that gives a finer-grained view of how the system behaves, and it helps reveal groups where performance differs from the total.

Slices can be drawn from many dimensions. Common examples include subscription tiers, such as paying users versus free users; traffic sources, such as mobile versus web; and usage patterns. Each of these splits can expose a group whose results differ from the total. Slicing only helps when you choose slices that matter to your users, so the choice of dimension is itself part of evaluation design.

One evaluation set rarely answers every question. Good practice is to maintain several sets, each serving a different purpose. One set should represent the distribution of actual production data, so you can estimate overall performance. Other sets should represent specific slices, so you can see how the system does on the cases that matter most. A common principle follows from this: if you care about something, put a test set on it.

Evaluation methods are techniques used to assess the performance of an application, and different criteria often need different methods. There is no single method that fits every question. The table below pairs common criteria with the methods that suit them.

Methods can also be mixed for the same criterion to balance cost and quality. A cheap classifier can give a lower-quality signal on all of your data, while an expensive AI judge gives a higher-quality signal on a small fraction, such as one percent. Together they provide a reasonable level of confidence while keeping costs manageable.

When log probabilities are available, they are worth using. Logprobs measure how confident a model is about a generated token, which is especially useful for classification. If the three class probabilities are each between 30 and 40 percent, the model is not confident in its prediction. If one class has a probability of 95 percent, the model is highly confident. Logprobs can also support measurements such as fluency and factual consistency for generated text.

Automatic metrics should be used as much as possible, but human evaluation remains valuable, including in production. Open-ended responses are hard to evaluate automatically, so human experts can review a subset of each day's outputs to detect changes in performance or unusual usage patterns. Evaluation in production also differs from experimentation: during experiments you may have reference data to compare against, while in production that reference data may not be immediately available, though real users' feedback is.

A criterion can only be scored if there are examples with reliable labels. Annotated evaluation data is built in stages. The steps below describe how examples move from raw material to a labeled evaluation set.

Annotation is needed for each component of your system and for each criterion. This applies whether the application is evaluated turn by turn in a conversation or as a whole task. The labeled data can later be reused, and the source notes that data curated and annotated for evaluation can be reused to synthesize more training data.

An evaluation set should be large enough that its result is reliable, but small enough to remain affordable to run. Bootstrapping offers a practical check. You draw repeated samples from your evaluation set and see how much the result moves. If the results swing widely across bootstraps, the set is too small to trust at its current size.

Bootstrapping tests reliability, not the truth of any single number. Stable bootstrap results suggest the set is large enough for this purpose. Sample size also matters when comparing two systems. If a new prompt scores 10 percent higher than the old one, a statistical significance test could in theory compute the sample size needed for a given confidence level, such as 95 percent, if you knew the score distribution. In practice that distribution is hard to know, so a rough guide is used instead.

That rough guide is a rule of thumb: for every threefold decrease in the score difference, the number of samples needed increases about tenfold. For reference, among benchmarks in Eleuther's lm-evaluation-harness, the median benchmark has 1,000 examples and the average has 2,159. The organizers of the Inverse Scaling prize suggested 300 examples as an absolute minimum and preferred at least 1,000, especially when examples are synthesized.

An AI application is only as good as the signals it receives about how it performs for real people. A model can be evaluated in a lab, but the true test happens when users type their requests, read the answers, and decide whether those answers helped. Those reactions are user feedback. User feedback is a crucial source of data for continuously improving AI models and applications.

Feedback serves three uses. Analytics tells you what is happening. Product improvement tells you what to change in the interface or the workflow. The data flywheel is the idea that data collected from usage can feed back into making the system better, which in turn can attract more usage and more data. This flywheel is one reason product experience has become a competitive advantage.

The diagram below sketches the path feedback takes from a user's action to a change in the product. It is a sketch of the general flow rather than a prescribed architecture. Follow it from left to right, then notice where the loop closes.

Feedback arrives in large volumes and in free-form language, which is hard to review by hand. The source notes that you can use AI to analyze feedback, not only collect feedback about AI applications. That is the step where raw material becomes something a team can act on. The source gives this idea only briefly and does not describe a particular method, so any specific technique for AI-assisted analysis would be an addition beyond what is established here.

Analysis then supports two kinds of decision. Analytics answers what is happening, and product improvement answers what should change in the interface or workflow. Both decisions can be checked against slices of users, so that a change that helps the majority is not shipped while quietly hurting a group that matters.

Conversational interfaces enable new types of user feedback. In a single-answer product, the main signal is a thumbs-up or thumbs-down on one output. In a conversation, the user is continuing an exchange. Their next message, their choice of wording, and the way they respond to a particular answer can all carry information about whether that answer was useful.

In the exchange above, the rephrased request may suggest that the first answer missed the point, and the decision to move on may suggest the revised answer worked. Neither signal is a rating, but both are informative. Feedback design in this setting is a shared responsibility. Traditionally it was treated as a product responsibility and often overlooked by engineers, but more AI engineers are now involved so that the system receives the data it needs.

Feedback is only useful if the application is built to collect it, so the way responses are presented matters. A direct design question is whether to show users full responses or to offer them options to choose from. The choice involves a trade-off, and opinions on it are divided. Both sides have arguments, so the choice should be made deliberately rather than by default.

Collecting feedback is harder for open-source applications than for commercial ones. The reason is structural. Users can take an open-source application and deploy it themselves, and the original developers have no visibility into how it is being used. Without that visibility, the usage data that drives analytics, product improvement, and the flywheel is much harder to gather.