179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Model Architecture· Hyperparameters
If dmodel = 8192 and I want SwiGLU, what is dff?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo calculate the feed-forward dimension (dff) for SwiGLU, you multiply the model dimension (dmodel) by 8/3. For a dmodel of 8192, this results in 21,845.3. This value is then rounded to 21,888, which is the nearest multiple of 128. This rounding is essential for hardware tile alignment, ensuring efficient computation on modern accelerators.