Why do outliers appear more in larger models?

The appearance of outlier channels is an empirical phenomenon linked to model scaling. As models grow, they develop specific feature dimensions that function as gating or routing mechanisms, often corresponding to specific attention heads or MLP neurons. These dimensions learn to carry high-magnitude signals to perform specialized tasks. This behavior is often associated with phase transitions observed in scaling laws, where the model’s internal representation becomes increasingly specialized to handle complex data patterns.