A colleague says our new model got 91% on MMLU. What fo...

To properly contextualize a high benchmark score like 91% on MMLU, you must ask for specific details regarding the evaluation protocol. Essential follow-up questions include identifying the prompt format and few-shot count, the specific version of the evaluation framework used, whether decontamination was applied and what the resulting removal rate was, and whether the model is a base model or an instruction-tuned model, as these variables can drastically change performance metrics.