← All questions
LLMOps & ProductionMedium· Monitoring· Training Health· Metrics

What training health metrics do you log beyond the loss...

The core set of metrics includes loss, pre-clip grad_norm, log_Z, loss_ema_ratio, and nan_frac. Each provides specific insight into training health and should have defined thresholds to trigger alerts or actions. When a run exhibits unexplained instability, add per-layer gradient norm histograms, sampled every 100 steps to limit overhead, to localize the specific layer causing the issue.

Code playground (Python)