179 questions
No questions match those filters.
← All questions
Training & Fine-tuningVery Hard· Adam Optimizer· Maximal Update Parameterization· Optimization Theory
μP is derived for SGD on a deep linear network. Why does the Adam rule differ?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansBecause Adam normalizes gradient elements, the effective operator norm of the weight update becomes O(η√nlnl-1) regardless of the raw gradient scale. When you re-derive the update-stability condition under this normalization, the optimal learning rate scaling factor becomes η ∝ 1/nin. While this was derived for linear networks, empirical evidence shows that nonlinearities and attention mechanisms follow the same scaling behavior.