Splitting Data into Train and Test Sets

A dataset is a collection of records that a model learns from, and it is easy to assume that a record appearing twice is just a little extra material. Repetition, however, changes what a model learns. The same record that shows up many times carries its pattern with much more weight than a record that appears once. This article follows how that weighting happens, how a copy can leak from the training side of a split into the test side, and how to decide what counts as a duplicate in the first place.

A dataset is a sample of some part of the world. When certain records are copied, the sample tilts toward them, so the pattern they carry looks more common than it really is. In the red pencil example, the red pencil entries take up a large share of the data, while every other item appears only once. The diagram below compares the situation before and after the repeated records are removed. Watch the red pencil entries, the pattern that changes.

Before the copies are removed, the red pencil pattern dominates what the model sees, and the model may treat red as a signal of high price. After the copies are removed, the red pencil appears once, like every other item, and no single pattern is overweighted. The model is still learning from the same kinds of items, but the sample no longer leans on one of them.

Models are typically judged on data they did not learn from. A train and test split exists to keep those two groups separate: the training set is what the model learns from, and the test set presents inputs it has not effectively seen. Duplication can break that separation. When duplicated data is split into training and test sets, one example may land in the training set while its duplicate lands in the test set.

Follow the path in the diagram. The original record is split, and one copy goes to training while its twin goes to testing. During training, the model meets the answer through the first copy. When the model is later scored on the test set, the second copy looks familiar, so the score reflects memory of an example rather than performance on new inputs.

The effect is not only a matter of evaluation. Several studies report a negative effect of duplicated training data on model performance. One Anthropic study cited in the source found that repeating 0.1% of the data 100 times could make an 800M parameter model perform like a 400M parameter model, even though the other 90% of training tokens remained unique.

Duplication can also cost resources even when it does not visibly hurt the final model. Repeated material consumes training time and compute without adding new information, so duplicates can waste resources even when the model looks acceptable.

When the data is a collection of documents, duplication usually takes one of three forms. Recognizing which form you are dealing with helps you decide how to look for it, because some duplications are harder to detect than others. A repeated paragraph inside one file and a quote scattered across many files call for different search strategies, so it helps to ask first where the copies are likely to live.

Read the diagram column by column. Each column names one form, and the two boxes inside it are the pair of text units being compared. The scope grows as you move from left to right in thinking about the search: a whole document is the largest unit, while a quote can be a small fragment that appears in many places.

Finding duplicates starts with a definition, and the definition is a choice. Nothing about two records makes them duplicates on its own; you decide how to compare them. Three decisions shape the result: the level at which you compare items, the rule for how closely they must match, and whether order matters.

The match rule and the order question are where many duplicate decisions go wrong. An exact match finds only identical items, while a threshold finds near-copies as well. Whether order matters is not a technical fact about the data; it depends on what you want the comparison to measure. A deduplication step built for one purpose may be inappropriate for another.

The flowchart below turns the three choices into an ordered sequence of checks. Given two records, you pick a comparison level and a match rule, then ask whether order matters. The answer to that last question decides which comparison runs, and that is the step where reordering the items can change the result.

The example shows why the order decision must be made deliberately. Nothing in the two records settles it. The same pair can be a duplicate for one purpose and distinct for another, so the definition belongs to the task you are preparing the data for.