Why does production LLM training use BF16 instead of FP16?

FP16 uses a 5-bit exponent, which restricts the maximum representable value to 65,504. In large-scale LLM training, gradient norms frequently exceed this limit, leading to overflow and training instability unless complex loss scaling techniques are applied. BF16 uses an 8-bit exponent, matching the dynamic range of FP32. This eliminates the risk of overflow without requiring loss scaling, while the reduced mantissa precision is empirically sufficient for weights, activations, and gradients in LLM training.