How does chunked prefill interact with the KV cache, an...

Chunked prefill breaks a large prompt into smaller chunks of size C. Between these chunks, the scheduler can pause the prefill process to insert generation steps for other active requests. The KV cache grows incrementally with each chunk, resulting in the same final memory footprint as a standard prefill.

The primary benefit is that long-prompt requests no longer monopolize the GPU for seconds at a time, which dramatically improves p99 inter-token latency for other users. The trade-off is an increase in total prefill FLOPs due to the need for partial attention recomputation across chunk boundaries, which can be tuned by adjusting the chunk size C.