179 questions
No questions match those filters.
← All questions
Training & Fine-tuningHard· Maximal Update Parameterization· Depth Scaling· Residual Connections
μP is usually validated for width scaling with fixed de...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen scaling depth, activations tend to grow at a rate of O(√L) across L layers. To maintain the stability conditions required by μP, production implementations must pair width rules with a depth-dependent correction, such as initializing the output projection of each sublayer to zero or weighting it by 1/√L. While the original μP papers focused on width, depth scaling remains an active area of engineering.