179 questions
No questions match those filters.
← All questions
Training & Fine-tuningEasy· Optimizer Memory· Adam· SGD
Why does Adam make memory worse than SGD?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansUsing BF16, SGD requires 4 bytes per parameter to store parameters and gradients. Adam, however, requires storing the first moment (4 bytes, FP32), the second moment (4 bytes, FP32), and master weights (4 bytes, FP32). This adds 12 bytes to the base 4 bytes, totaling 16 bytes per parameter. This 4x blow-up is entirely attributable to the optimizer’s state requirements.