179 questions
No questions match those filters.
← All questions
System DesignVery Hard· Large Scale Training· Pipeline Parallelism· Fault Tolerance
If you had to train a 1T dense model on GPUs today, wha...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFor a 1T dense model, I would configure TP=8, with PP sized to fit the model (likely 32 or more), and DP for remaining capacity. The primary concern is the pipeline bubble; with PP=32, the bubble ratio requires at least 128 micro-batches per replica to keep idle time under 20%, which strains the global batch budget. Secondary concerns include the time required for checkpointing and the impact of mid-step GPU failures at 32K+ scale. Mitigations include zero-bubble pipeline scheduling and robust fault-tolerant checkpointing.