A user reports high tail latency at peak traffic. Walk...

At scale, high tail latency is most commonly caused by the scheduler packing more requests per step to maintain throughput as QPS rises, which increases per-step latency and forces tail requests to wait. Confirm this behavior by analyzing the step-time histogram. To fix it, prioritize adding replicas to spread the load and reduce the effective batch size per replica, implement preemption to cap the maximum batch size, or use priority queues to ensure latency-sensitive requests are processed in smaller batches.