179 questions
No questions match those filters.
How would you estimate how many H100s you need to serve...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansEstimating hardware requirements begins with calculating the memory footprint of the model weights (e.g., 140 GB for a 70B model, requiring two H100s for tensor parallelism). Next, calculate the KV cache size (Cseq) based on the expected context length. With these, determine the maximum batch size (Bmax) per replica pair. Finally, compute the TPS per replica pair and divide the total target RPS (1,000) by this number to find the required number of H100s. Assumptions regarding average output and context lengths must be stated explicitly.