Why does RLHF outperform SFT for instruction following at scale?

SFT is bounded by the time and cost constraints of human generation. In contrast, pairwise preference collection allows for the exploitation of the generator-validator gap, which extracts more usable signal per annotation dollar. To prevent the degenerate outputs often associated with unconstrained reward maximization, the training objective includes a KL divergence penalty, which keeps the RL policy grounded in the original SFT distribution.