What is DPO, and what is its main advantage over PPO for preference alignment?

Direct Preference Optimization (DPO) works by reparameterizing the RLHF objective, showing that the optimal policy under a KL-constrained reward maximization objective can be expressed through policy log-ratios. This allows the reward to be defined implicitly. By training using a binary cross-entropy loss over preferred and rejected pairs, DPO removes the need for a separate value network and the complex rollout generation step required by PPO. This architectural simplification reduces the memory footprint significantly, requiring only the policy and reference models.