Why does RLHF fail for reasoning tasks?

RLHF fails for reasoning tasks because the reward model is a noisy, finite proxy for human preferences. The policy exploits the gap between this proxy and ground truth, causing reward scores to rise while real quality stagnates—a classic example of Goodhart’s law. Furthermore, human feedback is too expensive to generate online at the scale required for training reasoning capabilities.