179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Attention Mechanisms· Model Architecture
What happens if you double the number of attention head...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen you increase the number of heads to 64 while keeping the model dimension at 4096, the head dimension drops to 64. Although the total number of attention parameters remains unchanged, the reduction in per-head capacity outweighs the benefits of increased pattern diversity. Consequently, this configuration is generally avoided in practice at the 7B scale.