How would you operate a mixed AR-diffusion cluster to m...

Operating a mixed cluster requires aligning model strengths with workload characteristics. AR models should be prioritized for interactive, recall-heavy, and short-output tasks where latency is the primary SLA. Diffusion models should be reserved for long-output, throughput-sensitive bulk processing. By sizing each backend to its specific SLA—AR for P95 latency and diffusion for tokens/second—and rebalancing the allocation weekly as the request mix evolves, you can optimize the overall cost per quality point.