179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Sequence Parallelism· Communication· Efficiency
Why is sequence parallelism said to be free?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIn a ring-based implementation, the AllReduce and the ReduceScatter + AllGather pair move the same number of bytes per device. Sequence parallelism rearranges when the scatter happens but does not increase the total volume of data transferred.