Why is Chatbot Arena considered the gold standard for e...

Chatbot Arena is widely regarded as the gold standard because it relies on live human preference across real-world user prompts, utilizing an ELO system that accommodates new models effectively. However, it faces significant limitations. Goodhart’s law applies here, as providers have begun manipulating protocols to inflate ELO scores, a phenomenon documented in ‘The Leaderboard Illusion.’ Furthermore, there is a structural population bias toward technically literate English speakers, which limits the generalizability of the results to a broader global user base.