Your RLHF reward model score is going up every epoch bu...

When reward scores rise while human quality drops, the policy has likely drifted into a distribution where the reward model is no longer accurate. To fix this, you can apply a KL penalty to constrain policy drift, collect fresh preference data from the current policy to update the RM, or switch to an outcome-based reward in domains where correctness is verifiable to anchor the signal.