What are the limits of the diffusion serving cost advan...

The diffusion model’s cost advantage is not universal. First, for short outputs (L < 64), the per-step weight-sweep cost is not sufficiently amortized. Second, poor length prediction leads to padding waste that erodes memory savings. Third, unlike AR models, diffusion models cannot leverage prefix caching, which is detrimental for workloads with long, shared system prompts. Quantitatively, the diffusion backend becomes more cost-effective than AR when the number of refinement steps K is less than the output length L.