179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· PPO· Value Function· RLHF
What is the value model for in PPO, and can you remove it?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe value model is essential in PPO because it provides a baseline that reduces the variance of the policy gradient. Without this baseline, gradient variance scales with sequence length, making training highly unstable. While it is possible to remove the value model, doing so requires an alternative approach to variance reduction, such as the one used in GRPO. GRPO replaces the value baseline with a group mean reward, which requires the model to perform multiple rollouts per prompt to compute the baseline correctly.