179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· Loss Spikes· Gradient Clipping· Training Stability
What causes loss spikes during large-scale training, an...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansLoss spikes occur when logit values grow excessively large, causing the softmax function to saturate and leading to a collapse in gradient flow. Modern training stacks prevent this using several techniques: global gradient norm clipping (usually at 1.0), QK-norm to normalize queries and keys before dot products, and z-loss to penalize large logit magnitudes. Additionally, soft logit capping can be used to bound logits to a fixed range without the harsh effects of hard clipping.