Suppose you disaggregate prefill and decode onto separa...

Disaggregating prefill and decode allows engineers to match hardware to the specific workload characteristics. Prefill-optimized hardware benefits from high-FLOP-count GPUs (like H100 SXM), while decode-optimized hardware benefits from high memory bandwidth or a higher density of GPUs per parameter shard.

Beyond hardware efficiency, this architecture allows for independent scaling of compute pools based on the distribution of prompt lengths versus output lengths. Crucially, it eliminates head-of-line blocking, ensuring that a request with a very long prompt does not stall the generation steps of other active requests, thereby improving overall system responsiveness.