If you iteratively retrain the reward model on data fro...

Iterative retraining fits a fresh finite-sample estimator on the current policy’s distribution, which closes the distribution gap at that specific step. However, label noise in human annotations is irreducible. Because you are still fitting a noisy estimator, the RL algorithm will eventually find its edges given enough steps. This is why noise-free verifiable rewards are fundamentally different, as they lack the proxy gap that RLHF models are prone to exploit.