What goes wrong if you accidentally differentiate throu...

When πold is not detached and shares the same parameter tensor as πθ, the ratio πθ/πold effectively becomes 1. By the chain rule, the gradient of this ratio with respect to θ is zero, rendering the clipping mechanism useless. While the KL divergence term may still produce gradients that allow training to appear functional, the fundamental trust region guarantee of the algorithm is destroyed.