Why does vanilla policy gradient have high variance, an...

The gradient of the objective function is derived via the log-derivative trick as ∇θJ = E[R∇θ log πθ]. Subtracting a baseline B(s) that depends only on the state is bias-free because the expectation of the gradient of the log-probability over actions is zero. This technique effectively centers the rewards, reducing variance in the gradient estimate.