You have 64 experts on 16 GPUs. Training throughput is...

Even if per-expert utilization metrics appear balanced, device-level aggregation can be skewed if multiple high-traffic experts are co-located on the same GPU. This creates all-to-all communication hotspots. To resolve this, you should introduce an Ldevice loss coefficient (typically in the 10^-3 range) to penalize device-level imbalance. If using fine-grained experts, applying top-M device selection can structurally bound the worst-case inbound traffic per GPU.