← All questions
LLMOps & ProductionVery Hard· Latency· Cost Optimization· System Design

We are moving retrieval from a local index at 40 ms to...

The conflict centers on the ratio of retrieval cost to generation time. At 400ms, every-sentence retrieval adds significant latency (e.g., 4.8s for a twelve-sentence answer), making infrastructure’s demand for triggering correct. To address quality’s concern about wasted compute on discarded drafts, you should lower the trigger threshold and hard-cap the number of triggers per answer. The final deliverable should be a monitored trigger-rate band with an automatic fallback to question-only retrieval, as the trigger rate is a property of the query mix and will shift over time.

Code playground (Python)