179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Grouped-Query Attention· KV Cache· Serving Costs
Your architecture team wants to reduce key-value heads...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansBy reducing the number of key-value heads, the memory footprint of the KV cache per sequence is reduced by a factor of four. This allows the maximum batch size to roughly quadruple when memory is the limiting factor. Consequently, this cuts the cost-per-token at throughput-limited operating points while leaving the time-to-first-token essentially unchanged.