Logit soft-capping’s gradient through the tanh vanishes...

Once a logit exceeds approximately 2s, the gradient of the loss with respect to that logit is attenuated by (1 - tanh^2(l/s)), which is near zero. The optimizer receives almost no signal to reduce the logit. While the soft-cap prevents the softmax from seeing the extreme value, the underlying logit norm can continue to grow unchecked. This confirms that soft-capping addresses the symptom rather than the cause, whereas QK-norm prevents norm growth at the source.