179 questions
No questions match those filters.
← All questions
System DesignHard· Memory Budgeting· Hybrid Attention· Inference Scaling
How do you design a system to serve a 70B model at 128K...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansServing a 70B model at 128K context on two H100s requires careful memory management. After accounting for the model weights (approx. 140GB in bf16), roughly 20GB remains for the KV cache. By implementing hybrid attention—using sliding-window attention for local layers and global layers with GQA-8—the KV cache memory footprint can be reduced to under 5GB, making the deployment feasible within the hardware constraints.