179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Memory Estimation· Model Architecture
Roughly how much memory do the MLP weights consume in a...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo estimate the memory, consider that each layer contains three matrices of size 4096 x 11008. At 2 bytes per parameter (BF16), one matrix is approximately 90 MB. With three matrices per layer, that is 270 MB per layer. Across 32 layers, the total is approximately 8.6 GB. This aligns with the rule of thumb that MLPs and attention mechanisms each consume about half of the total model parameters.