What happens if you train a MoE without any load balanc...

If a MoE is trained without load balancing, router collapse occurs quickly, where the model relies on only 1-2 experts. The other experts receive no gradient signal and remain near random initialization. Consequently, the model effectively becomes a dense model of size K x dexpert, but it carries the ‘dead weight’ of the remaining N-K expert parameter sets, making it less efficient than a correctly-sized dense model with the same FLOP budget.