Why does RLHF sometimes make models worse even though t...

The reward model is a finite-sample proxy for human preference. As the policy optimizes against this proxy, it finds stylistic and structural patterns that score well on the proxy but were underrepresented in the reward model training, creating a train-test gap. Consequently, the proxy reward and the true win-rate diverge after a critical number of RL steps, an inflection point consistently observed across both human and noisy AI feedback settings.