Using Trusted Evaluators for Comparative Evaluation
Trusted evaluators are the approach used in a private comparative leaderboard.
Choosing Between Language Models
Suppose you want to know which of several language models is best. One option is to measure accuracy on a fixed test set. That works well for some questions, but many qualities that matter in a response are hard to capture that way. Tone, helpfulness, and whether a model should have refused a request at all are examples. A test set with a single correct answer cannot easily score any of these.
A different approach asks people directly. You show someone a prompt, show them two answers, and ask which answer is better. Repeat this across thousands of people and thousands of prompts, and the votes can be turned into a ranking of models. This article follows one vote through that process, explains why the resulting ranking is useful but imperfect, and looks at the main ways teams try to repair it.
From One Vote to a Model Ranking
Trace a single comparison through the process. A prompt is written by a person. Two models each produce a response, and the person sees both responses side by side. The person then votes for the answer they prefer. That vote is one data point. On its own it says only that this person, on this prompt, preferred this response. It says nothing yet about which model is stronger overall.
The ranking appears only when many such votes are combined. Each vote involves a pair of models, so the votes add up to a record of which model beat which, across many people and many prompts. The diagram below shows the path from one vote to the final ordering of models.
Notice what the single vote does and does not carry. It records a preference, not a proof. Whether the final ranking means anything depends on what those votes were measuring and whether the votes were collected under comparable conditions. The rest of this article examines those two questions.
Why Crowd Preference Is Useful but Imperfect
A common assumption holds that if enough ordinary people prefer one model's answers, that model is the better one. Human preference gathered from real use, sometimes called preference in the wild, feels like the most honest measure available because it reflects what people actually want. The approach has real strengths, but the assumption hides an important gap between preference and the purpose you care about.
Crowd preference is real signal, not automatic truth. Whether it is the right evidence depends on the use case.
Standardization: No Shared Meaning of Better
Standardization asks whether every vote is measuring the same thing. In an open crowd it usually is not. Anyone can submit any prompt and vote however they like, so there is no agreed definition of what makes one response better than another, and no mechanism that enforces one. The evaluation therefore lacks standardization.
The effect is that the same pair of responses can receive different verdicts depending on who judges it and under what conditions. The concept map below shows the factors that can push one pair toward different outcomes. None of them requires a voter to act dishonestly.
Quality Control: Who Votes and How Carefully
Quality control asks who is voting and how carefully they read. Open voting offers no mechanism that checks either. This produces two consequences that matter for ranking.
- First, voters may reward answers that sound more convincing even when they are wrong. Checking facts takes effort, and volunteers are unlikely to spend it on every comparison.
- Second, voters differ in what they want. Some prefer polite, moderate replies, while others prefer replies without filtering. Both preferences are real, and a single ranking cannot honor both unless someone decides which one counts.
Tracing a Crowd Ranking Under Two Prompt Pools
Same crowd, different prompt pools
Two teams run open crowd voting on the same two models. Team One collects a pool of very short, simple prompts. Team Two collects a pool of longer prompts that include retrieved documents, similar to what their product will send. Predict whether the two rankings must agree, then trace the reasoning.
- 1Identify what each vote measures
Each vote records which answer a person preferred for the prompt they were shown. The vote is only as representative as the prompt behind it.
- 2Compare the prompt pools
Team One's short prompts may reward brief, polished replies. Team Two's prompts with retrieved documents may reward answers that use the supplied material accurately. The two pools ask different questions.
- 3Check for a shared standard
Neither team has a predetermined standard for better, so each ranking reflects its own pool and its own voters. There is no common yardstick that forces the two results to match.
- 4Conclude about the rankings
The two rankings can disagree even though both were built honestly from crowd votes. Disagreement here reflects different inputs, not necessarily a mistake by either team.
The rankings need not agree. Each one is a valid record of preference over its own prompts, and neither automatically tells you how the models will perform inside a system that supplies retrieved documents.
Mistakes Teams Make With Crowd Rankings
Common mistake Treating any crowd preference as the correct answer
What it looks likeA model wins most open votes, so the team assumes it is the best model for their product.
Why it failsCrowd preference is real signal, but its fit depends on the use case. The votes measure what voters liked, which may not match your goal.
Do this insteadAsk what the votes actually measure and whether that matches the quality your system needs.
Common mistake Using a leaderboard without checking its prompts
What it looks likeA team picks a model from a leaderboard built on simple, repeated prompts, then deploys it in a system that retrieves documents for every request.
Why it failsSimple or repeated prompts can pollute a ranking, and leaderboards that omit real-world context may not reflect performance inside your system.
Do this insteadInspect how the prompts were chosen and whether they resemble your real inputs before relying on the ranking.
Common mistake Assuming voters check facts
What it looks likeA confident but incorrect answer keeps winning votes.
Why it failsVoters may reward answers that sound more convincing, because checking facts takes effort volunteers are unlikely to spend.
Do this insteadTreat open votes as measures of perceived quality unless a separate check confirms correctness.
Common mistake Mixing rankings from different criteria as if they were comparable
Four Fixes and What Each One Gives Up
Once the problems are visible, the natural question is how to restore standardization without losing what made the crowd useful. Four main approaches are described: predetermined prompts, trusted evaluators, in-product comparisons, and AI evaluators. Each enforces some standard, and each introduces a cost. No option removes the tension entirely.
Predetermined prompts are the most direct way to enforce a standard, but they work against breadth. A related alternative lets users submit any prompt and then filters out the hard ones with an internal model, ranking models only on the hard subset. That keeps diversity in the collection step while controlling what the ranking is built from.
Trusted evaluators are the approach used in a private comparative leaderboard. Evaluators are trained on the criteria for comparing two responses, or trained to use practical prompts and sophisticated prompting techniques. The consistency is valuable, but the cost is real, and the number of comparisons you can collect shrinks.
Embedding comparisons in a product moves evaluation into real work. In a code editor, for example, the product can show a user two code snippets and ask which one they prefer. The benefit is grounding in a real workflow. The weakness is that an ordinary user may not be the expert needed to judge the snippets, and some users will click without reading. Even so, the votes of the careful minority can sometimes be enough to show which model is better.
AI evaluators replace human judges with a model. Some teams prefer this because a model may be more consistent than random internet users, even though it may not match a trained human expert. This is presented as a trade-off rather than a clean fix, so the choice depends on how much consistency you need compared with how much expertise.
| Approach | Standard it enforces | Main cost | What it gives up |
|---|---|---|---|
| Predetermined prompts | A fixed set of prompts defines what is measured | Narrower coverage of real use | Breadth of prompts and tastes |
| Trusted evaluators | Training on comparison criteria and practical prompting methods | Expense, and fewer comparisons collected | Volume of votes |
| In-product comparisons | Real workflow context | Users may lack expertise or may click without reading | Expert judgment and careful reading |
| AI evaluators | Consistency from a model judge | May not match a trained human expert | Human expertise |
Each fix enforces some standard and introduces a cost, so the right choice depends on your goal.
Choose the trade-off that matches your goal, and disclose it. A ranking published with its prompts, evaluators, and collection method lets readers judge whether it fits their purpose.
Practice: Matching a Fix to a Goal
A team wants to rank two models for a customer support product. Customers ask questions that require answers drawn from a retrieved help-article database. Which fix would you choose first, and what does your choice give up? Explain your answer in terms of standardization, quality control, coverage, and cost.
Hints
- Think about whether the team needs their real inputs, which include retrieved documents, to be represented in the evaluation.
- Consider whether the team needs a consistent standard and how expensive it is to collect comparisons with trained evaluators.
Key Takeaways From the Vote-to-Ranking Path
- 1A single A vs. B vote records one person's preference on one prompt. A model ranking appears only when many votes across people and prompts are combined.
- 2Crowd preference is useful and hard to game, but it is real signal rather than automatic truth. Its fit depends on the use case.
- 3Open crowdsourcing lacks standardization and quality control, so the same response pair can receive different verdicts from different raters, and honest voters can still produce a distorted ranking.
- 4Predetermined prompts, trusted evaluators, in-product comparisons, and AI evaluators each enforce some standard at a cost. Choose the trade-off that matches your goal and disclose it.
At a glance
- A single A vs. B vote records one person's preference on one prompt; a model ranking emerges only when many votes across people and prompts are combined.
- Crowd preference is useful and difficult to game, but it is real signal rather than automatic truth, and its fit depends on the use case.
- Open crowdsourcing lacks a shared standard for better and lacks quality control over who votes and how carefully, so the same pair can receive different verdicts.
- Predetermined prompts, trusted evaluators, in-product comparisons, and AI evaluators each enforce some standard at a cost; choose the trade-off that matches your goal and disclose it.