How does FlashAttention’s backward interact with sequen...

When using sequence parallelism like ring attention, each device holds a contiguous shard of queries and computes partial gradients for K and V based on remote tiles passed around the ring. After the ring completes, each device has the full dQ, but only partial dK and dV. An all-reduce operation is required to aggregate these partial gradients. To maintain performance, this all-reduce communication is typically overlapped with the forward pass of the next micro-batch to hide the latency of the network transfer.