Labeling Interfaces for Evaluation Work

Picture a team building an AI system for a customer. The engineers have started drafting an evaluation set, the collection of examples used to judge whether the system behaves well. A customer contact then says the domain experts are busy this week and will try to find time later. The team shrugs and keeps working, because they have the technical side covered. This is a common way of thinking, and the source material pushes against it.

Keep your answer in mind. The rest of this article tests it against what the source says about co-created evaluation sets, and it ends by asking you to apply that reasoning to a case of your own.

The central claim of the source is direct: an evaluation set should be co-created with the customer, not written by the development team alone. The engineers can draft examples, but the standard that the set measures against should come partly from the people who know the domain. The source names those people as the customer's domain experts.

The diagram below traces how knowledge and labeled examples move through a co-created set. Read it from the left: the experts' judgment enters through the labeling work, the labels become part of the set, and the development team contributes to the same finished result. The figure shows only the relationships the source states. It does not show every stage of a real project.

The source's reason for involving customers is that their domain experts know what good looks like. That knowledge is the thing the evaluation set is trying to capture. An engineer can write an example of an input, but deciding whether a real output is acceptable, and explaining why, depends on expertise that lives with the customer.

The source is specific about how the experts should take part. They must be part of building the evals, they should be involved early, and they should be hands-on. It also says the labeling interface should let them work efficiently. The experts are therefore not reviewers who look over a finished set at the end. They are builders who label examples while the set is being made.

Three conditions appear together in the source: early involvement, hands-on work, and an interface that makes the work efficient. Early involvement means the judgment shapes the set from the start. Hands-on work means the experts act on actual examples. An efficient interface means their time goes into judging rather than fighting the tool.

A team could try to collect expert knowledge through a questionnaire asking the customer to describe their requirements. The source says this approach has a limit: the knowledge is difficult to extract through a survey. Stated requirements can tell the team what the customer wants in general. They are much less able to show how the customer's judgment works when an actual output is in front of them.

The judgment that matters is the one an expert applies when looking at a real output, deciding whether it is acceptable, and explaining why. That is a different kind of act from answering a questionnaire, and it is the reason the source asks for experts to take part directly. The comparison below sets the two approaches side by side.

Now follow the state of the evaluation set as the engagement moves forward. Each stage changes what the set contains and who has shaped it. The steps below are drawn from the source's description of co-creation, with the labeling session as the point where expert judgment enters.

The source sets a concrete expectation: the first eval-labeling session should happen in the first week of validation. If an hour of a domain expert's time cannot be secured during that week, the source concludes the engagement lacks the customer's attention it needs.

This changes how the missing hour should be read. It is not treated as a neutral calendar event. It is treated as evidence about participation. The reasoning is that co-creation depends on the experts being hands-on and early, so an expert who cannot appear in the first week is a sign that the customer's involvement is not yet where the work needs it to be.

The source does not say exactly what the team should do next after the signal appears. Its claim is narrower: the absence in the first week is evidence that the customer's attention is lacking. A careful reader should take that claim as stated and avoid adding steps the source does not describe.