179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Mixture of Experts· Ablation Studies
How would you run the ablation to decide whether to inc...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo properly decide on including a shared expert, you should train three 1B-parameter models for 100B tokens each, keeping hyperparameters fixed. The variants are: (a) 0 shared, 64 routed, K=6; (b) 1 shared, 64 routed, K=6 (adding 17% FFN FLOPs); and (c) 0 shared, 64 routed, K=7 (adding 17% FFN FLOPs via an extra routed expert). Comparing held-out perplexity against the FLOP budget determines the winner. If (b) beats (c), the shared expert is justified; if (c) beats (b), adding a routed expert is a more efficient use of compute.