Data Contamination in Public Benchmarks

A public leaderboard answers a specific question: which model performed best on that leaderboard’s tests? Your application poses a different question: which model performs best on the tasks your product actually needs? A model ranked first overall may not be the best fit when your application depends heavily on only a few capabilities.

Treat model selection as building a private leaderboard. Start from the work your application does, choose benchmarks that test the capabilities it needs, and check that each benchmark measures what its name suggests. Public benchmark results can help you shortlist models, but they do not replace more rigorous experiments using your own benchmarks and metrics.

A useful comparison begins with a benchmark set chosen for your application, not with whichever public ranking is easiest to find. Reliability matters: anyone can publish a benchmark, and some benchmarks may not measure what their names suggest. Public scores also have practical limitations. Model scores can change over time, some models have no published score for a benchmark, and running evaluations yourself can be costly.

Score drift is not merely a hypothetical concern. The source describes a study that found significant changes in GPT-3.5 and GPT-4 performance on many benchmarks between March 2023 and June 2023. The practical lesson is to treat a score as a measurement from a particular evaluation, not as a permanent description of a model. When published scores are absent, an evaluation harness can help organize running a model against a benchmark and collecting the results.

Once you have collected scores, decide how important each benchmark is to your application. Weighting makes that priority explicit: a benchmark central to the application should count more than one that is only loosely related. Because raw benchmark scores may use different units or scales, do not add them as though they were directly comparable. First make sure the scores you combine are on a basis that supports comparison, then apply your chosen importance weights.

A high public benchmark score does not by itself establish that a model has learned the capability the benchmark is intended to test. The score assumes that the model was not trained on the benchmark’s questions or answers. If those items were in the training data, the model may reproduce answers it has encountered. This is data contamination, also called data leakage or training on the test set.

A satirical 2023 paper made the risk vivid: its author trained a model with one million parameters exclusively on data drawn from several benchmarks. The model reached near-perfect scores on those benchmarks and outperformed much larger models on them. The example illustrates why benchmark performance can reflect exposure to evaluation material rather than the broader ability an evaluator hoped to measure.

The route is often indirect. Models may be trained on text collected from the internet, and the collection process may include a benchmark published online before training. Benchmark material can also spread through shared source material or copied files. In each case, the training process can overlap with later evaluation even when nobody deliberately selected the test items. Deliberate inclusion can happen too: a developer may use a benchmark to choose a model and later continue training on that benchmark’s data. That may be a decision the developer chooses to make, but it still means the released model is contaminated for evaluation on those items.

Two heuristics can help estimate whether evaluation samples are present in, or familiar to, a model’s training data: n-gram overlapping and perplexity. These are screening approaches, not proof that a benchmark is clean or contaminated. The source characterizes n-gram overlapping as more accurate but costly and dependent on having training data available. It characterizes perplexity as cheaper but less accurate.

For n-gram overlapping, compare benchmark samples with the training data to look for overlapping text. This route is useful only when the training data is available, and its greater accuracy comes with higher cost. Perplexity offers a cheaper screen, but its lower accuracy means its result should not be treated as conclusive. The source provides no universal cutoff for either heuristic, so do not invent one: interpret screening results in context and be clear about the method’s limits.

Handling contamination has two broad parts: detect it, then decontaminate where practical. One approach is to remove evaluation samples from training data. That is not always possible: many users of foundation models do not control the training data. Even developers who do control it may decide not to remove every benchmark sample, because high-quality benchmark data can help model performance. New benchmarks can also appear after a model has already been trained, so some contamination will remain possible.

When removing benchmark samples is not practical, report the percentage of benchmark data that appears in the training data and provide performance on both the full benchmark and the clean samples. Showing both results helps readers see whether the result changes when contaminated examples are excluded. The source’s comparison of GPT-3 results illustrates why the effect should not be assumed to move in one direction or by one amount: changes differed across benchmarks, and a result based on a very small clean sample can be noisy.

Other checks can add context to a reported score. Public leaderboard hosts such as Hugging Face plot the standard deviation of model performance on a benchmark, helping outliers stand out. Well-designed public benchmarks may also keep part of their data private and evaluate models against that held-out set. These practices do not remove the need to disclose contamination, but they can make comparisons more informative.