Even with a clean protocol, what is the fundamental limit of human preference evaluation for assessing model quality?

The fundamental limit of human preference is that it captures user sentiment rather than objective correctness. In tasks involving math, code, or factual retrieval, a confident but incorrect answer is often preferred by humans over a hedged but correct one. Consequently, a single ELO score is misleading because it collapses subjective and objective performance. The solution is to use human preference exclusively for subjective tasks like tone and style, while relying on ground-truth metrics for objective, verifiable tasks.