Why is autoregressive generation memory-limited rather than compute-limited?

Autoregressive generation is memory-limited because, at a batch size of 1, the arithmetic intensity is far below the compute saturation point of modern GPUs like the H100. Specifically, MLP arithmetic intensity is roughly B FLOP/byte and attention is 1 FLOP/byte, while the H100 requires a much higher intensity to reach the compute ridge point. Increasing the batch size allows for linear efficiency gains until the KV cache consumes the available HBM, at which point throughput saturates.