The attention arithmetic intensity is always ≈ 1 no mat...

To address the low arithmetic intensity of attention, two primary strategies exist. First, reduce the size of the KV cache to minimize the bytes transferred per step. Techniques like Grouped Query Attention (GQA) reduce the number of KV heads, Multi-head Latent Attention (MLA) projects the KV cache into a lower-dimensional latent space, and sliding-window attention discards cache entries beyond a specific window.

Second, one can move away from autoregressive generation entirely. Diffusion-based language models generate all tokens in parallel and refine them iteratively. This shifts the bottleneck from memory bandwidth to compute, yielding significant throughput improvements at the cost of more complex sampling dynamics.