How does the number of KV heads interact with tensor pa...

In a tensor-parallel (TP) architecture, the KV cache must be distributed across the participating GPUs. Because each GPU requires at least one KV head to function, the KV head count acts as a constraint on the maximum effective TP degree. If the TP degree is higher than the number of KV heads, the system is forced to replicate KV heads across GPUs. To avoid this inefficiency, a best practice is to design the KV head count as a multiple of the largest anticipated TP degree.