179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Latency· ColBERTv2· Performance Engineering
Your ColBERTv2 index fits in memory, but latency is hig...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansHigh latency in ColBERTv2 is often a result of excessive data movement during the scoring phase. By calculating the total bytes moved per query, you can identify the bottleneck. The structural fix involves using the centroid indices to perform an initial pruning pass. This allows you to decompress only the most promising candidates, drastically reducing the amount of data processed and improving p95 latency.