179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Tensor Parallelism· GQA· System Design
At inference time, how does the GQA KV head count inter...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIn a tensor-parallel inference setup, each shard must hold complete KV heads to function correctly. If the number of KV heads (HKV) is not divisible by the TP degree, the system is forced to replicate KV heads across shards. This replication negates the cache savings intended by GQA. A robust design ensures that HKV is a multiple of the maximum anticipated TP degree.