179 questions
No questions match those filters.
What’s the difference between TTFT and inter-token late...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTTFT (Time To First Token) scales with prompt length and model size, while ITL (Inter-Token Latency) scales with model weight bytes and batch KV cache size. In a streaming chat product, ITL dominates the user experience; if the text appears sluggish, the user perceives the system as slow.
While a 200ms TTFT is generally invisible to users, a 100ms ITL creates a perceptible lag. Therefore, optimization strategies differ: ITL is best improved via quantization and KV cache reduction, while TTFT is best addressed through multi-GPU prefill or prompt pruning.