179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Performance Tuning· Distributed Training
What is the communication overhead of AllReduce for a 7...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFor a 70B-parameter model using BF16 (2 bytes per parameter), the total size is 140 GB. The per-rank cost of AllReduce is approximately 280 GB. Given a 200 GB/s inter-node InfiniBand connection, the synchronization time is roughly 1.4 seconds per step. This overhead is generally unacceptable for high-performance training, necessitating techniques such as gradient compression, pipeline parallelism, or switching to ReduceScatter-based ZeRO.