Why does Z(x) drifting cause specific problems in bfloa...

Fp32 has 23 mantissa bits, providing approximately 7 decimal digits of precision, while bfloat16 has only 7 mantissa bits, providing approximately 3 decimal digits. When Z(x) is large (e.g., 10^7) and a token’s probability is very small (e.g., 10^-8), fp32 can represent this as a denormal nonzero value, but bfloat16 rounds it to exact zero. While the log-sum-exp trick prevents overflow in computing log Z, it does not prevent underflow in the resulting probabilities. Consequently, the model receives zero gradients for those tokens and loses the signal needed to refine predictions.