What can go wrong with reward model training and how do...

The two main failure modes in reward model training are length bias and distribution shift. Length bias occurs when the reward model assigns higher scores to longer responses regardless of their actual quality; this can be detected by measuring the correlation of reward model scores with response length across a set of neutral prompts. Distribution shift occurs during RL training when the reward model is forced to extrapolate to outputs it was not trained on. This is detected by monitoring the divergence between the reward model’s scores and human evaluation scores as the RL training process progresses.