179 questions
No questions match those filters.
← All questions
System DesignHard· Context Length· Memory Requirements· Scaling
A customer wants to extend context from 8K to 128K toke...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansExtending context to 128K tokens creates a massive memory requirement for the KV cache. For a model like Llama-2-13B, the KV cache alone can exceed 100 GB, which is larger than the entire HBM of a single H100. Consequently, it is impossible to serve a single 128K sequence on one GPU. The system must implement ring attention or sequence parallelism to distribute the sequence across multiple GPUs, which significantly increases the hardware and serving costs.