179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Attention Mechanism· Transformer Architecture
Walk me through what one attention head computes, and t...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe computation is defined by the attention pattern A = softmax(XWQW > K X>/√dh) and the output h = AXWV WO. These projections are defined only up to an invertible matrix R inside the head, meaning only the products WQK and WOV are intrinsic properties of the model. The attention pattern A mixes positions, while the WOV circuit mixes channels, making these two circuits genuinely independent rather than merely conceptually separable.