179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· Maximal Update Parameterization· Hyperparameter Tuning· Scaling
What is μP and what problem does it solve?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansμP ensures that activation magnitudes and gradient update sizes remain Θ(1) per coordinate regardless of the model width. By choosing specific initialization scales (1/√nin) and per-layer Adam learning rates (proportional to 1/nin), optimal hyperparameters discovered on a small proxy model can be transferred to a much larger scale without the need for additional re-tuning.