Removing Duplicates and Contamination from Datasets

A natural instinct when training a model is to collect as much data as possible, on the reasoning that more examples means more learning. The evidence reported in the source material points somewhere different. A modest collection of well-prepared examples can outperform a very large collection that is full of noise. In this context, noise means material that is irrelevant to the job the model must do, or material that contradicts itself. This article traces how a dataset is judged, step by step, and how duplicates and contamination are removed along the way.

High-quality data is judged against the task a model is being trained to perform. Quality is not a single number that a dataset carries on its own. Different people will disagree about the details depending on what they are building, so the practical question is always what good data looks like for this particular job. The source material does give a common starting checklist of six characteristics, which are covered in the next section.

The same example can be high quality for one task and low quality for another. The source illustrates this with legal material. If the task is to answer legal questions as they apply today, a legal dataset from the 19th century might not be relevant. If the task is instead about the legal system of the 19th century, that same dataset becomes highly relevant. Relevance is therefore not a property a document has in isolation. A document is relevant or irrelevant only in relation to what the model is supposed to do.

The source gives a common checklist: data can be considered high quality if it has six characteristics. It should be relevant, aligned with task requirements, consistent, correctly formatted, unique, and compliant. The table below summarizes the question each characteristic asks of a dataset.

Consider the source's example of scraped product reviews. Those pages often carry HTML tags around the review text. The tags are not part of what the review says, so they should be stripped before the text becomes training data. The worked example below follows a single raw record through the cleaning steps. The record and its copy are invented here for illustration, but the cleaning rules come directly from the source.

Alignment concerns the annotations, meaning the labels or target answers attached to each example. The principle is that annotations should match what the task requires. The source gives contrasting cases. If the task requires factual consistency, the annotations should be factually correct. If the task requires creativity, the annotations should be creative. If the task asks for a score along with a justification, each annotation should contain both. If the task asks for concise answers, the annotations should be concise.

The source deliberately uses the word aligned rather than accurate or correct. An accurate or correct response might not be what a user wants. A reply that is technically right but far too long for a quick question has not met the requirement. Alignment means matching the purpose of the task, not just being true.

Consistency asks whether the labels hold steady. Two properties matter. First, the same kind of example should receive the same kind of label. Second, different annotators working on the same example should produce labels that are not far apart. The source gives a direct test: if two annotators annotate the same example, their annotations should not be too different.

The source raises a practical question using essay scoring on a scale from 1 to 5. If two essays both receive a 4, are they really of the same quality? When an annotation scheme is vague, identical labels can hide very different judgments. The source states the consequence plainly: inconsistent annotations can confuse the model and make it harder to learn. The remedy it points to is a good annotation guideline, which gives annotators a shared standard so that the same label means the same judgment. Such a guideline is essential for annotations that are both aligned with the task and consistent with each other.

Uniqueness means the examples in a dataset are sufficiently unique from one another. The source is careful with the word sufficiently. Duplications can introduce biases and cause data contamination during training, but specific use cases can tolerate different levels of duplication. Uniqueness is therefore not a requirement of zero repeats. It is a level you decide on based on what the project can tolerate.

The reasoning behind the concern is that a repeated example carries extra weight. If one item appears many times, the model sees its pattern more often than the pattern deserves, which can skew what it learns. Contamination is a related danger: material that should be kept separate from the data used to judge the model can leak into training. Duplicates are one common route for that leak. The diagram below traces how an evaluation example can end up in training data through a repeated copy.

In the diagram, the evaluation example is meant to stay in the evaluation split. A second copy collected from the scraped pool then flows into the training split, so the model has effectively seen the material it will later be judged on. Removing that repeat is what keeps the two sets separate. Deciding how much duplication to allow is a judgment call tied to the goal of the model.

Correct formatting means that every example follows the format the model is expected to work with. The source adds a specific claim: redundant formatting tokens can interfere with the model's learning, so they should be removed. HTML tags wrapped around review text are the obvious case, but the source also warns about less visible issues. Trailing whitespace, stray new lines, inconsistent casing, and mixed numerical formats all count as formatting problems.

The final characteristic is compliance. Data should comply with all relevant internal and external policies, and the source explicitly includes laws and regulations in that category. The source's example is direct: if you are not allowed to use personally identifiable information to train your models, then your data should not contain any PII at all.

The source describes a project called LIMA, which fine-tuned a 65-billion-parameter Llama model on 1,000 carefully curated prompts and responses. Human annotators judged its answers equivalent to or strictly better than GPT-4's answers in 43 percent of cases. The same source also notes a tradeoff: a model trained on that few examples was not as robust as product-grade models. A small, curated set can therefore be very effective for some goals while still falling short on others.