Writing Annotation Guidelines

The source material gives a working definition: data is high-quality when it helps you do your job efficiently and reliably. The obvious follow-up question is what that looks like in a concrete dataset. The answer is not a single number. People building different systems will disagree about the details, because the right data depends on what the system is supposed to do. The source does, however, offer a common starting checklist of six characteristics. Once you see how the same example can pass one test and fail another, the rest of this article becomes easier to apply.

A natural instinct when training a model is to collect as much data as possible, since more examples sounds like more learning. The evidence reported in the source points somewhere different. A modest collection of well-prepared examples can outperform a very large collection that is full of noise. In this context, noise means material that is irrelevant to the job, or material that contradicts itself.

The source gives two concrete cases. The creators of the Yi model family reported that around 10,000 carefully crafted instructions were superior to several hundred thousand noisy ones. A separate project, LIMA, fine-tuned a 65-billion-parameter Llama model on 1,000 carefully curated prompts and responses. Human annotators judged its answers equivalent to or strictly better than GPT-4's answers in 43 percent of cases. The source also notes a tradeoff: a model trained on that few examples was not as robust as product-grade models. So small curated sets are not a free replacement for scale in every situation, but they show that the preparation of examples matters as much as their count.

The first two characteristics both ask whether the data fits the job. Relevance asks whether an example belongs to the task at hand. As the previous section showed, relevance is never a fixed property of a document. It only exists in relation to what the model is supposed to do. The source's legal example makes this concrete: a 19th-century legal dataset may be irrelevant to a task about today's law, but highly relevant to a task about 19th-century legal systems.

Alignment concerns the annotations, meaning the labels or target answers attached to each example. The principle is that annotations should match what the task requires. The source contrasts several cases. If the task requires factual consistency, the annotations should be factually correct. If the task requires creativity, the annotations should be creative. If the task asks for a score along with a justification, each annotation should contain both. If the task asks for concise answers, the annotations should be concise.

The source uses the word aligned deliberately, instead of accurate or correct. An answer can be technically right and still fail the task. A correct but very long reply to a quick question has not met the requirement. Alignment means matching the purpose of the task, not merely being true.

Consistency asks whether the labels hold steady. Two properties matter. First, the same kind of example should receive the same kind of label. Second, different annotators working on the same example should produce labels that are not far apart. The source gives a direct test: if two annotators annotate the same example, their annotations should not be too different.

The source raises a practical question using essay scoring on a scale from 1 to 5. If two essays both receive a 4, are they really of the same quality? When an annotation scheme is vague, identical labels can hide very different judgments. The source states the consequence plainly: inconsistent annotations can confuse the model and make it harder to learn. The remedy is a good annotation guideline, which the source treats as essential for annotations that are both aligned with the task and consistent with each other. In practice, this means that a guideline should say what each label means, so that the same label means the same judgment to every annotator.

Correct formatting means that every example follows the format the model is expected to work with. The source makes a specific claim: redundant formatting tokens can interfere with the model's learning, so they should be removed. Its example is scraped product reviews, which often carry HTML tags around the review text. Those tags are not part of what the review says, and they should be stripped before the text becomes training data. The source also flags less visible problems: trailing whitespace, stray new lines, inconsistent casing, and mixed numerical formats. None of these changes the meaning of a review to a human reader, which is why they are easy to overlook. A model, however, sees each variation as a difference it has to account for.

Uniqueness means the examples in a dataset are sufficiently unique from one another. The source is careful with the word sufficiently. Duplications can introduce biases and cause data contamination during training, but different use cases tolerate different levels of duplication. Uniqueness is therefore not a demand for zero repeats. It is a level you decide on based on what the project can tolerate. The reasoning is that a repeated example carries extra weight: if one item appears many times, the model sees its pattern more often than that pattern deserves, which can skew what it learns. Contamination is a related danger, where material that should be kept separate from the data used to judge the model leaks into training. Duplicates are one common route for that leak.

The final characteristic is compliance. Data should comply with all relevant internal and external policies, and the source explicitly includes laws and regulations in that category. Its example is direct: if you are not allowed to use personally identifiable information to train your models, your data should not contain any PII at all. The source adds two closing notes that apply to all six characteristics. The first is that the six are a default checklist, and specific use cases may add their own requirements. The second is that these checks belong at the planning stage. Before creating data, think about what each characteristic means for your project, because the techniques used to produce data are aimed at producing exactly these qualities.