How do you handle a query that needs information from multiple documents?

A single retrieval call only surfaces documents relevant to the query as a whole — it can’t answer something that genuinely requires combining facts scattered across separate documents. The standard fix is query decomposition: break the question into its component sub-questions, retrieve separately for each, and let the model combine the results. This works because each sub-question is now a normal single-hop retrieval the system already handles well.

Two other patterns exist for harder cases. Iterative retrieval lets the result of the first search shape what gets searched for next, useful when the sub-questions aren’t obvious upfront. GraphRAG goes further and builds an explicit knowledge graph connecting entities across the corpus, which pays off specifically when the relationships between things — not just the facts themselves — are what the query is asking about.

A finance bot asked to compare Q3 2024 revenue against Q3 2025 revenue can’t answer that from one search, since no single chunk contains both numbers. Splitting it into two separate retrievals — one per quarter — then handing both results to the model for comparison took accuracy on that class of question from around 40% to over 90%.