179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· RLHF· Reward Hacking· Optimization
Why can’t you just do gradient ascent directly on the reward model output?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansReward models are imperfect, finite-capacity proxies for human preference. If you perform direct gradient ascent on the reward model output, the policy will quickly exploit the reward model’s blind spots, leading to reward hacking where the model achieves high scores while producing low-quality content. To prevent this, you must use a KL penalty and a clipped surrogate loss, which constrain the policy and prevent it from moving into regions of the output space that have high reward model scores but poor actual quality.