How do you detect and mitigate benchmark contamination...

Contamination happens when benchmark examples appear in pretraining data, either directly or through near-duplicates, allowing a model to score high via retrieval. Detection strategies include calculating n-gram overlap between benchmark test sets and pretraining corpora, performing membership inference attacks to see if the model assigns higher likelihood to test examples than held-out examples, and using canary insertion to check if the model recalls synthetic labeled examples. The primary mitigation strategy is data-side: deduplicating pretraining data against known benchmarks before the training process begins.