← All questions
System DesignHard· Latency· RAG· Optimization

The p99 budget just dropped from 800ms to 250ms. One en...

The decision depends on the measured value of the selector. If the selector’s contribution to accuracy is near zero, zero-shot is the correct choice. If it is high, reducing the number of chunks may hurt recall. A third option is to replace the heavy selector with a distilled 110M-parameter scorer, which can maintain performance while meeting the tighter latency requirements.

Code playground (Python)