179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Activation Functions· Model Architecture
Why did LLaMA switch from GeLU to SwiGLU?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe switch to SwiGLU is driven by the addition of a learned per-dimension gate that selectively suppresses hidden activations. Ablation studies across multiple research efforts demonstrate consistent perplexity gains for the same parameter and FLOP budget. The choice is further validated by Shazeer’s original GLU paper and subsequent adoption in models like PaLM and T5. When implementing, an 8/3 factor is typically applied to the hidden dimension to account for parameter consequences.