← All questions
Training & Fine-tuningVery Hard· Adam Optimizer· Maximal Update Parameterization· Optimization Theory

μP is derived for SGD on a deep linear network. Why does the Adam rule differ?

Because Adam normalizes gradient elements, the effective operator norm of the weight update becomes O(η√nlnl-1) regardless of the raw gradient scale. When you re-derive the update-stability condition under this normalization, the optimal learning rate scaling factor becomes η ∝ 1/nin. While this was derived for linear networks, empirical evidence shows that nonlinearities and attention mechanisms follow the same scaling behavior.

Code playground (Python)