If your RAG endpoint's p50 latency is too high, where s...

Optimizing a RAG system requires identifying the dominant cost. In systems where the context runs to several thousand tokens, the decode phase is almost always the bottleneck. Because decode is memory-bandwidth-bound and occurs for every output token, it typically consumes the vast majority of the latency budget. Efforts should be directed toward reducing decode time through quantization, batching, or shortening output lengths, rather than optimizing minor stages like reranking or vector indexing.