← All questions
Inference & ServingHard· KV-cache· Attention· Accuracy

After shipping chunk-level KV-cache reuse, accuracy reg...

When chunks are cached in isolation, the model loses the ability to perform cross-document attention, which is essential for comparison tasks. Single-fact lookups remain unaffected because they only require information from a single chunk. A strong answer proposes checking fine-tune coverage on multi-document examples or exempting compare-style queries from the cache path. Blaming position-id reassignment is a common trap, as that would cause ordering confusion rather than specific comparison failures.

Code playground (Python)