How do you detect and prevent reward over-optimization...

To manage reward over-optimization, maintain a separate evaluation signal not used for training, such as a held-out reward model, automated benchmarks, or human A/B testing. Monitor these signals continuously throughout the training run. Additionally, use KL regularization to limit policy drift, and stop the training process when the independent evaluation signal plateaus or drops.