179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Model Architecture· FFN
What inner FFN dimension should be used for a 7B model...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFor a 7B model with a dmodel of 4096 using SwiGLU, the inner FFN dimension is typically set to 8/3 of the dmodel. This calculation yields 10923, which is rounded to 11008 to ensure it is a multiple of 128. It is important to adjust this ratio compared to standard ReLU-based FFNs to avoid inflating the parameter count by 50%.