Why is autoregressive decoding slow compared to prefill?

Prefill is compute-bound because it processes all tokens in parallel, allowing the system to utilize tensor cores effectively. In contrast, autoregressive decoding generates tokens sequentially, resulting in very low arithmetic intensity for both projections and attention mechanisms regardless of sequence length. Because the compute requirements are low per token, the process is limited by the speed at which the GPU can move data from HBM, making it memory-bound rather than compute-bound.