179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Transformer Architecture· MLP· Model Scaling
Why is the MLP parameter count so sensitive to the dff...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe MLP parameter count is sensitive because the transition from ReLU to SwiGLU introduces an additional weight matrix. To keep the total parameter budget consistent with previous architectures, the hidden dimension (dff) must be adjusted downward. Naively adopting SwiGLU without scaling the hidden dimension would result in a significant, unintended inflation of the model’s parameter count by roughly 50%.