How does DDP overlap gradient communication with the ba...

DDP optimizes synchronization by grouping parameters into buckets (defaulting to 25 MB). As the backward pass computes gradients, the hook for the last parameter in a bucket triggers an asynchronous all_reduce for that entire bucket. This allows communication to occur concurrently with the computation of earlier-layer gradients. However, this overlap is limited because the final bucket on the critical path must complete its all_reduce before the optimizer step can begin. If a model has very few parameters per layer, the buckets may serialize, causing the overlap to collapse.