What makes distributed inference across multiple GPUs s...

Single-GPU serving only has to deal with the prefill/decode asymmetry within one device: prefill is compute-bound and processes the whole prompt in parallel, decode is memory-bandwidth-bound and produces one token at a time. Multi-GPU serving inherits that same asymmetry and adds a second axis of difficulty on top of it: communication.

Sharding a model across GPUs — whether by tensor parallelism (splitting individual layers) or pipeline parallelism (splitting layers across devices) — requires synchronizing activations or gradients across GPUs at every layer boundary, over an interconnect that’s orders of magnitude slower than on-chip memory bandwidth. That communication cost is roughly constant per token, but decode’s per-step compute is already tiny, so the fixed communication overhead eats a much larger fraction of each decode step’s budget than of each prefill step’s — making decode disproportionately sensitive to how the model is sharded and how fast the interconnect is.

On top of that, a real cluster is serving a mix of requests at different phases and different sequence lengths simultaneously, so naive scheduling lets a GPU sit compute-idle waiting on a decode step for one request while a prefill-heavy request queues elsewhere. This is exactly why modern serving stacks (like disaggregated prefill/decode architectures) increasingly route prefill and decode to physically separate GPU pools tuned for their respective bottleneck, rather than forcing every GPU to handle both phases well at once.