What does reranking actually add to a RAG system that i...

The initial vector search step uses what’s called a bi-encoder: the query and every chunk get embedded completely independently, and similarity is just a distance calculation between two vectors that never actually saw each other. That’s what makes it fast enough to search millions of chunks, but it’s also approximate — a chunk can look superficially similar in embedding space to a query while a semantically closer chunk sits a bit further away purely as an artifact of how bi-encoder embeddings compress meaning.

A reranker is a cross-encoder: it reads the query and one candidate chunk together, in the same forward pass, and scores their relevance directly rather than comparing two independently-computed vectors. That joint reading catches relevance the bi-encoder missed, but it’s far too slow to run against an entire corpus — which is exactly why the standard pattern is retrieve broad and cheap (say the top 20-50 by vector similarity), then rerank that shortlist down to the 3-5 that actually go into the prompt. A chunk that ranked third by cosine similarity routinely jumps to first after reranking, and this single addition is usually the highest-ROI change available to an existing RAG pipeline.