The search team wants LLM query expansion on every requ...

The choice between query-time expansion and index-time document expansion (doc2query) is primarily a trade-off in latency and operational flexibility. While compute costs for both approaches often balance out over a short period, index-time expansion adds zero latency to the p99, whereas query-time generation consumes a significant portion of the 250ms budget. If the corpus churns faster than the re-generation job, index-time expansion becomes impractical. A hybrid approach—expanding documents offline, caching expansions for head queries, and using online generation for tail queries—is the most robust design.