What is contextual compression in RAG, and is it actual...

Even a well-retrieved, well-reranked chunk is usually mostly irrelevant to the specific question — it got retrieved because one sentence or phrase inside it matched, not because the surrounding paragraph is useful. Contextual compression addresses that by extracting just the portion of each chunk that actually bears on the query, typically with a dedicated LLM call or a purpose-built extractor, and discarding the rest before it ever reaches the generation prompt.

The payoff is threefold and it compounds. Fewer tokens in the prompt means lower per-query cost, directly. Less irrelevant surrounding text means less noise the model has to filter past to find the answer, which measurably helps accuracy. And the token budget that compression frees up can be spent on including more distinct sources in the same context window rather than fewer, bulkier ones — a chunk about a company’s founding history and unrelated office locations can compress from 450 tokens down to the 80 that actually state a file-size limit, an 80%+ reduction with no loss in answer quality.