179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Parallelism· Memory Management
Walk me through your parallelism strategy for a 70B mod...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansStart by calculating the static memory footprint, which includes the 70B parameters (at 2 bytes each) plus optimizer states. Set Tensor Parallelism (TP) to 8 to match the node size. Then, choose Pipeline Parallelism (PP) such that each GPU’s memory capacity is respected. Use the remaining GPUs for Data Parallelism (DP). Finally, calculate the resulting global batch size and verify it against the critical batch size to ensure training stability, while also monitoring the pipeline bubble fraction.