What is reward overoptimization, and how can it be dete...

Reward overoptimization, or reward hacking, occurs when a policy finds outputs that achieve high scores under the reward model but are not actually preferred by humans. To detect this, one should track gold human preference win-rates alongside the reward model scores; a divergence between these two metrics serves as a clear signal of overoptimization. Mitigation strategies include increasing the KL regularization coefficient, reducing the number of PPO steps per batch, or employing an ensemble of reward models to make it significantly harder for the policy to find and exploit weaknesses in the reward function.