Our p99 retrieval is 240 ms and we need 150. The vendor...

Do not assume the vector database is the problem. Attribute the latency budget to the four primary candidates: the embedding call, network latency, the reranker, and the round trip to fetch payloads. Rank these by their actual impact; for example, a cross-encoder can consume over 100 ms, and cross-region network hops can add significant overhead. Switching vector databases is rarely the solution if the search component only accounts for a tiny fraction of the total latency.