Where would you look first if users were reporting high...

The first step in diagnosing high latency is to determine whether the issue is with Time to First Token (TTFT) or per-token latency. High TTFT suggests a prefill bottleneck, which may be caused by prompts that are too long, batch sizes that are too large, or insufficient compute provisioning. Conversely, high per-token latency points to a decode bottleneck, often caused by KV cache pressure, memory bandwidth saturation, or batch sizes that exceed the capacity of the memory hierarchy.