179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· ZeRO· FSDP· Communication Overhead
What is the communication cost of ZeRO stage 3 compared...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe communication cost difference between ZeRO stage 3 and stage 1 is defined by the parameter sharding strategy. ZeRO stage 3 requires an extra P of communication because it performs a per-layer parameter AllGather during the forward pass to materialize the weights needed for computation. This represents a 1.5x increase in communication volume, which is the trade-off required to achieve N times parameter memory savings by sharding parameters across all ranks.