A model you trained gets 91% on MMLU. How do you know i...

To verify that a model has genuinely learned the material rather than memorizing the test set, you must first run 13-gram decontamination to identify and remove overlapping sequences between the training and test data, verifying that the removal rate is non-zero. Additionally, you should run an ordering probe on the evaluated model. If the score drops significantly under shuffled-choice ordering, it is a strong indicator that the model was relying on memorized patterns rather than reasoning, confirming contamination.