We need 3× throughput improvement on a production 7B mo...

To achieve a 3x throughput improvement, we must address the bandwidth-bound nature of autoregressive generation. By quantizing weights to INT4 using AWQ, we reduce the amount of data that must be moved from memory to the compute units by 4x compared to BF16. AWQ is specifically chosen because it uses activation magnitudes to guide the quantization, which preserves model accuracy much better than naive weight-only quantization. This approach typically yields 2-3x real-world speedups with minimal impact on benchmark performance.