What are the three main failure modes of token-choice top-K routing in MoE models, and how does auxiliary-loss-free load balancing address them?

The three primary failure modes are: load collapse, where all tokens route to a single expert; token dropping, which occurs when an expert exceeds its capacity; and routing instability, characterized by oscillations during early training. Auxiliary-loss-free load balancing mitigates these by maintaining a running Exponential Moving Average (EMA) of expert load. It then adjusts routing scores to favor underloaded experts, effectively balancing the load without the need for an auxiliary loss term that might otherwise compete with the primary training objective.