Your 70B pre-training run develops a loss spike at 150B...

First, check if the grad_norm shows a single-step extreme outlier, which suggests a data anomaly, or a gradual elevation, which points to logit drift or learning rate issues. If log_Z has been rising for thousands of steps, the root cause is logit scale instability; in this case, roll back to the checkpoint before the acceleration, add z-loss with alpha = 10^-4, and resume. If these metrics do not explain the spike, inspect the data shard window for corrupted documents or pipeline errors that altered the data distribution.