179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· MoE· Routing· Efficiency
Why not use a soft gate over all experts instead of top-K?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIf you were to use a soft gate over all experts, you would be forced to compute the FFN output for every single expert for every token. This would multiply the FFN FLOP cost by the number of experts, N. Consequently, you would end up with a dense model containing N FFNs, losing any efficiency advantage that MoE is designed to provide. Top-K is the critical mechanism that keeps the FLOP budget constrained to the level of a dense baseline while allowing the model to scale its total parameter count.