179 questions
No questions match those filters.
← All questions
Inference & ServingHard· FlashAttention· Tiling· Kernel Optimization
Walk me through one tile of the FlashAttention forward...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIn a single tile of the FlashAttention forward pass, the kernel loads Qi, Kj, and Vj into SRAM. It computes the attention scores Sij = QiKTj / sqrt(d). It then updates the running maximum m’ = max(m, rowmax(Sij)) and the running normalizer l’ = l * exp(m - m’) + rowsum(exp(Sij - m’)). The output accumulator O’ is updated using the rescaled values, and the process repeats for the next KV tile. After all tiles are processed, the final output is obtained by dividing O by l.