Limiting Users to Predetermined Prompts
A user is shown a prompt and two responses, one from each model being compared, and is asked which response is better.
Why Ask People Instead of Testing
What do you think happens?
Suppose you want to know which of several language models is best. Would you rely only on a fixed test set with right and wrong answers, or would you ask people which response they prefer? Make your guess before reading on.
Reveal answer
Many teams ask people directly, showing a prompt and two answers and asking which answer is better.
A fixed test set can measure accuracy, but many useful qualities of a response, such as tone, helpfulness, or whether the model should have refused a request at all, are hard to capture that way. Asking people directly, across thousands of people and thousands of prompts, lets the votes be turned into a ranking.
This article follows one idea through the whole process: a crowd's votes are real evidence, but the meaning of those votes depends on what was asked and how the votes were collected. Crowdsourced comparison is attractive because it gathers many tastes and many prompts. Its weakness is that the votes are not controlled, so a single ranking can hide disagreements that matter for your purpose.
Following One Vote Through the Pipeline
A crowdsourced ranking is built from many small events. A user is shown a prompt and two responses, one from each model being compared, and is asked which response is better. That single choice is one vote. The votes are then collected and turned into a ranking of the models. Tracing one vote from the interface to the final ranking shows where the evaluation can go wrong.
Notice where the quality problems can enter. The prompt is chosen freely, so different voters may be comparing responses to different questions. The vote itself depends on whether the user reads carefully. The collection step gathers whatever votes arrive, so unless something filters them, weak votes count the same as careful ones. Only after this does the tally turn individual choices into a single ordering of the models.
Two Failure Modes in Open Votes
The crowdsourced approach has real strengths. It collects a wide range of signals, because many people bring many tastes and many prompts. It is also relatively hard to game, since a single actor cannot easily fabricate a large, varied crowd. The same openness, however, creates two problems. Anyone can submit any prompt and vote however they like, so there is no agreed definition of what makes one response better, and no mechanism that enforces one. The evaluation therefore lacks both standardization and quality control.
The two problems are separate, so it helps to name them apart before looking at fixes.
Four Ways to Restore a Standard
Once the problems are visible, the natural question is how to restore standardization without losing what made the crowd useful. Each approach enforces some standard, and each introduces a cost. No option removes the tension entirely, so the practical task is to choose the trade-off that matches your goal and to disclose it.
Predetermined prompts are the most direct way to enforce a standard. Every voter answers the same fixed prompt set, so the win rates share a baseline. The cost is breadth: the collection is narrower than what an open crowd would produce. A related alternative lets users submit any prompt and then filters out the easy ones with an internal model, ranking models only on the hard subset. That keeps diversity in the collection step while controlling what the ranking is built from.
Trusted evaluators are the approach used in a private comparative leaderboard. Evaluators are trained on the criteria for comparing two responses, or trained to use practical prompts and sophisticated prompting techniques. The consistency is valuable, but the cost is real, and the number of comparisons you can collect shrinks.
In-product comparisons move the evaluation into real work. In a code editor, for example, the product can show a user two code snippets and ask which one they prefer. The benefit is grounding in a real workflow. The weakness is that an ordinary user may not be the expert needed to judge the snippets, and some users will click without reading. The source notes that the votes of the careful minority can sometimes be enough to show which model is better.
AI evaluators replace human judges with a model. Some teams prefer this because a model may be more consistent than random internet users, even though it may not match a trained human expert. The source presents this as a trade-off rather than a clean fix, so the choice depends on how much consistency you need compared with how much expertise.
Comparing the Trade-offs
Reading across the four approaches shows the pattern. Predetermined prompts and trusted evaluators both buy standardization, but predetermined prompts limit what is asked, while trusted evaluators limit how many judgments you can afford. In-product comparisons keep the scale and the real-world context, but they give up control over who votes and how carefully. AI evaluators give consistency at scale, but they inherit the question of whether that consistency matches the judgment you actually need.
Worked Example: Two Rankings That Disagree
Why two rankings from the same crowd can disagree
Two rankings are produced from the same open crowd in two different months. In the first month, Model X ranks above Model Y. In the second, Model Y ranks above Model X. Nothing about the models changed. What explains the reversal, and what does it tell you about using the ranking?
- 1Check what each vote was about
In an open crowd, each user chose their own prompt. The prompt mix in the two months may have differed, with one month including more hard questions and the other more casual ones.
- 2Check whether the prompts share a baseline
If the prompt sets differ, the win rates describe different tests. A model that wins on casual prompts may lose on hard ones, so the two rankings are not measuring the same thing.
- 3Check what the voters valued
Some voters prefer polite, moderate replies and others prefer replies without filtering. If the voter mix shifted between months, the ranking can move even though the models did not.
- 4Decide what the ranking can support
The reversal does not show that either model is stronger for your purpose. It shows that the ranking depends on what was asked and who answered. For a decision, you would need a fixed prompt set that matches your use case.
The reversal comes from changes in the input, not from a change in model strength. A ranking from an open crowd is only meaningful for the prompts and voters that produced it.
Mistakes Teams Make With Crowd Rankings
Common mistake Treating the crowd's ranking as the single true answer about which model is best
What it looks likeA team sees Model X at the top of an open leaderboard and switches its product to Model X without checking anything else.
Why it failsThe votes are real but their meaning depends on the prompts and voters that produced them. The ranking may not reflect how a model performs in your system.
Do this insteadTreat the ranking as one signal, and check it against the kind of work your product actually does.
Common mistake Assuming that simple or repeated prompts say something useful about hard use cases
What it looks likeA leaderboard is filled with easy prompts that many users repeat.
Why it failsSimple or repeated prompts can pollute a ranking, so the results describe easy cases rather than the cases that matter.
Do this insteadLook at the prompt mix, or use a predetermined prompt set that covers the difficulty you need.
Common mistake Ignoring real-world context such as retrieved documents
Why it failsA leaderboard that omits context may not reflect how a model will perform once it is inside your system, where it may receive retrieved documents alongside the prompt.
Do this insteadEvaluate with the context your system actually supplies.
Common mistake Choosing an evaluation method without disclosing its trade-off
Choosing a Fix for Your Goal
A small product team wants to compare two language models for a customer support assistant. Their main worry is that open user votes would mostly reward confident-sounding answers, even when those answers contain errors. They have a limited budget for expert reviewers. Which of the four fixes would you recommend first, and what trade-off would you disclose to the team?
Hints
- Think about which fix gives a shared standard across all votes.
- Consider what the team loses when the number of comparisons shrinks.
- Ask whether the reviewers need to check facts, and whether an AI evaluator is a clean substitute for that.
Article Summary
- 1A crowd's preference is a useful ranking signal, but it is not automatic truth, and its fit depends on the use case.
- 2Open crowdsourcing gives wide, hard-to-game signals, but it lacks a shared standard, so standardization and quality control are difficult.
- 3Predetermined prompts, trusted evaluators, in-product comparisons, and AI evaluators each enforce some standard while adding a cost, so the right choice depends on your goal, and the trade-off should be disclosed.
At a glance
- A crowd's preference is a useful but imperfect signal: the votes are real, but what they measure depends on the prompts and voters involved.
- Open crowdsourcing lacks standardization, because no shared standard defines what counts as better, and it lacks quality control over who votes and how carefully.
- Two common problems are rankings built on different prompts, which have no shared baseline, and voters who reward convincing but wrong answers.
- Predetermined prompts, trusted evaluators, in-product comparisons, and AI evaluators each enforce some standard with a cost, so choose the trade-off that matches your goal and disclose it.