GRPO maintains an old policy for importance sampling. H...

In GRPO, the importance-sampling ratio (pi_theta / pi_old) is critical for stability. If rollouts are reused for too many gradient steps, this ratio drifts outside the typical PPO clip range (epsilon approximately 0.2), causing the variance of the gradients to explode. While GRPO applies clipping at the group level to manage this, the trade-off for extending reuse is either a slower convergence rate due to a reduced learning rate or the need to widen the clipping threshold, which can introduce its own instabilities.