179 questions
No questions match those filters.
← All questions
Inference & ServingVery Hard· Memory Budget· Tensor Parallelism· KV Cache
Walk through the full memory budget for serving Llama-2...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFor Llama-2 70B in bf16, model weights require 140GB, which is 70GB per GPU when split across two. The KV cache calculation (2 * 80 * 64 * 128 * 2048 * 2 * 32) results in approximately 68GB. Total memory per GPU is 104GB, exceeding the 80GB H100 limit. Solutions include sharding the KV cache via sequence parallelism, reducing the batch size to approximately 14, or using GQA to shrink the KV cache footprint.