Why is perplexity on a held-out set a poor proxy for downstream task performance, and what should you measure instead?

Perplexity is a measure of a model’s ability to predict the next token averaged over a reference corpus. It is inherently sensitive to tokenization, domain distribution, and corpus quality rather than specific task capabilities. Consequently, a model with lower perplexity can still perform poorly on benchmarks like MMLU or HumanEval if the pretraining data is high-quality but mismatched to the evaluation task. For alignment-stage evaluation, metrics such as win-rate against a reference model or human preference judgments—often using GPT-4 as a proxy judge—are better indicators of real-world utility.