When does a dense model win at inference even if the Mo...

In memory-bandwidth-bound serving, such as single-user or small-batch scenarios, the hardware must stream all model weights from HBM for every token. A large-total-parameter MoE streams significantly more bytes than a smaller dense model, causing higher latency. MoEs only become competitive at large batch sizes where the hardware becomes arithmetic-bound, allowing the MoE to leverage its higher quality at the same activated-parameter count.