If rank 2 throws a Python exception mid-training, what...

In a distributed training job, collective operations are synchronous across all ranks. If one rank fails or throws an exception, the remaining ranks will block indefinitely at the next collective call, waiting for the failed rank to participate. To recover, the system must either explicitly handle the exception by broadcasting an error signal and calling dist.destroy_process_group() to clean up, or rely on an external job manager to identify the dead process and initiate a full job restart.