What’s wrong with using only Chatbot Arena or MMLU as your evaluation?

Relying solely on Chatbot Arena or MMLU is insufficient for production-grade evaluation. Chatbot Arena suffers from selection bias, as the user base is skewed toward tech-savvy individuals rather than the general population. MMLU is a quizzing distribution that rarely resembles the actual tasks a model performs in a deployed environment. While these benchmarks serve as useful leading indicators, they cannot replace the insights gained from production telemetry.