179 questions
No questions match those filters.
← All questions
System DesignHard· Pipeline Parallelism· Cluster Topology· Distributed Training
A 400B+ training run spans multiple rack spine switches...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen training massive models across multiple racks, physical placement is critical. Pipeline Parallelism (PP) requires frequent point-to-point activation transfers, which should be kept on high-bandwidth leaf switch fabric by grouping consecutive stages in adjacent nodes. Data Parallelism (DP) involves gradient all-reduces that occur once per step and are more tolerant of the higher latency associated with traversing spine switches. Ignoring rack-locality is a common cause of unexplained throughput variance.