179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· Distributed Training· NCCL· CUDA
Walk me through what happens when your training loop ca...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen dist.all_reduce() is invoked, the Python API triggers the torch.distributed NCCL shim. This shim enqueues a specific CUDA kernel to the GPU. The GPU then performs a ring-allreduce operation using high-speed interconnects like NVLink to synchronize the data across all participating ranks. Once the kernel is launched, the CPU is no longer involved in the data path, and the result is written back in-place on all ranks.