179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Attention Mechanisms· Model Architecture
For a 7B model with dmodel = 4096, how many attention h...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansUsing 32 heads for a 4096-dimensional model provides a head dimension of 128. This is the optimal configuration because it keeps the total attention parameters constant at 4 × dmodel^2 per layer. This specific setup is validated by the LLaMA-2 7B architecture, which demonstrates that this balance between head count and head dimension is effective for performance.