179 questions
No questions match those filters.
← All questions
Inference & ServingMedium· Latency· Cost Optimization· Self-RAG
You enabled Self-RAG. p50 latency moved four percent and the GPU bill tripled. Explain.
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansBecause the K candidate continuations are independent, they run as one batch. Since decoding is memory-bandwidth-bound, batching K candidates costs roughly one step of wall clock time, explaining the minimal latency impact. However, the bill measures total tokens processed, and since you are generating K times the content, the compute cost triples.