179 questions
No questions match those filters.
← All questions
Inference & ServingHard· Reranking· KV-cache· Causal Masking
A team builds a causal decoder reranker with the query...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe team failed to realize that in a causal decoder, tokens attend to previous tokens. By putting the query first, the document tokens are forced to attend to the query, which makes the document’s key and value vectors dependent on the specific query. To enable caching, the document must be processed in a way that remains independent of the query, which is defeated by this specific ordering.