179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· ZeRO· Distributed Training· DDP
Why is ZeRO stage 1 claimed to have zero communication...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIn standard DDP, the AllReduce operation is used to synchronize gradients across ranks. ZeRO stage 1 leverages the fact that AllReduce internally decomposes into a ReduceScatter followed by an AllGather. By inserting the per-rank optimizer step between these two phases, ZeRO stage 1 shards the optimizer states across ranks. Because the total bandwidth remains 2P, the optimizer state sharding is effectively achieved at no additional communication cost compared to standard DDP.