179 questions
No questions match those filters.
← All questions
System DesignHard· Prefix Caching· KV Cache· Retrieval
Your tech lead wants to delete the retrieval service: t...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe lead’s proposal ignores the physical memory requirements: 800k tokens at 131 kB results in 105 GB of KV cache, which exceeds the 80 GB capacity of modern GPUs. Furthermore, the system is brittle; any edit to a single article invalidates the cache from that point forward, forcing a full re-prefill. A better architecture keeps a cached long-context path for global synthesis while routing standard queries through a retrieval system.