Compare QK-norm and soft-capping. When would you choose...

QK-norm is generally preferred because it addresses the root cause, adds minimal parameters, improves perplexity, and enables higher learning rates. Soft-capping is simpler because it requires no learned parameters and can be retrofitted into existing implementations without changing projection layers. However, soft-capping can worsen perplexity due to tanh distortion of logit ordering. Choose QK-norm for greenfield runs and soft-capping if you need to stabilize an existing architecture with minimal code changes.