179 questions
No questions match those filters.
Why is LLM generation slow? What’s the bottleneck?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansLLM generation is limited by memory bandwidth because each step involves matrix-vector multiplications and KV cache reads that have an arithmetic intensity near O(1). This is far below the roofline performance of modern GPUs like the H100.
Because the GPU must load the entire model weight matrix and the KV cache from HBM for every single token generated, the performance is constrained by how fast the hardware can move data. The tensor cores are often underutilized because they finish the math faster than the memory system can provide the necessary data.