← All questions
RAGVery Hard· Latency· RL· SLO· Optimization

Research wants a learned per-query stopper. Platform wants a hard cap of two rounds, because p95 is a contractual SLO and variance is the enemy. Adjudicate.

A fixed cap optimizes the tail of the distribution, while a learned stopper optimizes the mean but may widen the tail. If more than 15% of traffic requires three or more hops, a learned stopper will not help the p95. Furthermore, learned stoppers introduce OOD risks. The best approach is to use the data from an offline sweep to determine the optimal static cap and use prefix caching to handle deeper queries without violating the SLO.

Code playground (Python)