179 questions
No questions match those filters.
You’re serving a 70B model and a user sends a 50k-token...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe TTFT for a large prompt can be estimated using the formula: 2 * Parameters * SequenceLength / ComputeCapacity. For a 70B model and a 50k-token prompt on an H100 (approx. 989 TFLOPS), the calculation is 2 * 70B * 50,000 / 989T, resulting in roughly 7 seconds.
This latency is unacceptable for interactive applications. To mitigate this, one should distribute the prefill workload across multiple GPUs using pipeline or tensor parallelism, implement chunked prefill with a draft model for the initial context, or apply prompt compression techniques to reduce the input size before it reaches the model.