← All questions
Inference & ServingHard· FlashAttention· Tiling· Kernel Optimization

Walk me through one tile of the FlashAttention forward...

In a single tile of the FlashAttention forward pass, the kernel loads Qi, Kj, and Vj into SRAM. It computes the attention scores Sij = QiKTj / sqrt(d). It then updates the running maximum m’ = max(m, rowmax(Sij)) and the running normalizer l’ = l * exp(m - m’) + rowsum(exp(Sij - m’)). The output accumulator O’ is updated using the rescaled values, and the process repeats for the next KV tile. After all tiles are processed, the final output is obtained by dividing O by l.

Code playground (Python)