How do you decide whether to roll back, adjust hyperpar...

A senior approach distinguishes between recoverable spikes and unrecoverable divergence. A spike is considered recoverable if the gradient norm returns to baseline within 50-100 steps and the loss follows suit. If the loss does not recover within 200 steps, or if the norm remains elevated and the loss plateaus above the pre-spike level, the model has diverged and should be rolled back to the last stable checkpoint. If the spike is recoverable, it is important to log the specific step and batch index for future contamination investigation.