179 questions
No questions match those filters.
If prefill is compute-limited and generation is memory-...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen a large prefill request arrives, it consumes significant GPU compute resources, which can block the memory-bound generation tasks of other requests. This causes noticeable spikes in inter-token latency (ITL). To mitigate this, chunked prefill limits the number of prefill tokens processed in a single step, allowing the GPU to interleave prefill and generation. Tools like FlashAttention-2 support this via the varlen interface, which handles mixed sequences in a single kernel call, slightly increasing TTFT while significantly improving ITL stability.