What is ColBERT's late-interaction mechanism, and which...

A standard bi-encoder pools every token vector from its encoder into a single fixed-size embedding per side (mean or [CLS]), which means every token contributes equally to a document’s identity regardless of relevance — a stopword and the one load-bearing entity get the same weight. ColBERT refuses that pooling: both query and document keep every per-token vector, projected to a smaller dimension and L2-normalized, and scoring is MaxSim — each query token independently finds its single best-matching document token, and the score sums those maxima across query tokens. Because the only cross-side comparison happens after both sides are already encoded, document representations stay precomputable and indexable, unlike a true cross-encoder where the representation depends on the query.

What the published ablations actually show is instructive: collapsing to a single [CLS] vector costs by far the largest chunk of accuracy; replacing the max with a mean costs much less; dropping query-side expansion costs almost nothing. So the granularity of keeping per-token vectors — not the max operator itself — is doing most of the work. It’s also worth knowing that a well-trained single-vector retriever with good hard-negative mining recovers much of that gap at a fraction of the storage, so multi-vector retrieval is a real accuracy/storage trade-off, not a strictly dominant architecture.