GRPO drops PPO’s value network. What replaces it, and w...

In GRPO, the value network is removed to save memory. The baseline is derived from the mean reward of G rollouts for a given prompt, and the advantage is the z-score of the reward within that group. The memory cost of a critic at 7B is 112 GB (14 GB weights, 14 GB gradients, 84 GB AdamW state). The trade-off is the compute cost of running G rollouts per question. A critical failure mode exists: the group baseline is only effective when the pass rate is not at the extremes of 0 or 1, because if all G rollouts agree, the critic signals nothing.