Why does a Mixture-of-Experts (MoE) model offer better...

In a dense model, every parameter is used for every token. In contrast, an MoE model routes each token to the K highest-scoring experts out of E total experts. This allows the model to maintain a small active-parameter count per token while utilizing a much larger total parameter count. Because the MoE has E/K more unique parameters than a dense model at the same active-FLOPs budget, it can encode a more complex function space, resulting in superior loss performance.