How would you measure the actual allreduce bandwidth av...

To measure actual allreduce bandwidth, use the NCCL all_reduce_perf benchmark. By sweeping message sizes from 1 MB to 10 GB, you can identify two distinct performance regimes. Messages below approximately 100 MB are latency-bound, while larger messages are bandwidth-bound. The peak bandwidth-bound value obtained from this benchmark is the appropriate metric for estimating training step times. If an NVSwitch chip fails, the remaining chips reroute traffic, reducing effective crossbar bandwidth by roughly 25% and increasing allreduce latency, which can degrade peak throughput by 20-30%.