What is an activation outlier and why does it break nai...

Activation outliers occur when certain dimensions of the residual stream grow to magnitudes 100 times larger than typical values, a phenomenon that is amplified by model depth and scale. Naive per-tensor quantization is ineffective because the quantization scale is dominated by these outliers, causing smaller, significant activations to be rounded to zero. SmoothQuant addresses this by migrating the scaling difficulty from activations to weights, multiplying the outlier dimension in activations by 1/s and the corresponding weight row by s before quantization occurs.