Why does KV cache fragmentation hurt throughput?

When memory is fragmented, the system cannot pack as many sequences into the GPU’s memory as it could with contiguous allocation. Since LLM inference, particularly in the MLP layers, is heavily memory-bound, the system relies on high arithmetic intensity achieved through large batch sizes. When fragmentation forces a reduction in the number of concurrent sequences, the GPU spends more time waiting for data from memory rather than performing computations, causing throughput to drop proportionally to the reduction in batch size.