What goes wrong when you naively quantize a large trans...

In large transformers, a small subset of channels exhibits extreme activation magnitudes. When using a standard absmax quantization approach, these outliers force the quantization scale to be very large, which results in significant precision loss for the majority of the activations that have smaller magnitudes. This ‘coarsening’ of the quantization grid leads to accuracy degradation. The standard fix involves isolating these outlier channels and processing them in higher precision while keeping the rest of the model in INT8.