179 questions
No questions match those filters.
← All questions
System DesignHard· Context Window· Parallelism· Ring Attention
How would you design a training system for 128K context?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWithin an 8-GPU NVLink node, apply TP=8 for weight sharding and SP=8 for LayerNorm sharding. For attention, apply CP=8, rotating KV in a ring while each GPU holds 16K queries and uses online-softmax tile accumulation. Pipeline parallelism handles model depth across nodes. The CP ring adds roughly one AllGather-equivalent of communication per layer, which can be masked by overlapping ring sends with computation.