Why do you need to call torch.cuda.synchronize() before...

In PyTorch, operations on the GPU are asynchronous. When you call a function like dist.all_reduce(), the CPU merely submits the command to the GPU’s work queue and returns immediately. If you stop a timer immediately after the function call without calling torch.cuda.synchronize(), you are measuring how long it took to submit the task to the queue, not how long the GPU took to perform the actual computation or communication.