← All questions
Training & Fine-tuningHard· LoRA· Generative Retrieval· Catastrophic Forgetting

Our proposal is a nightly LoRA adapter trained on the d...

While LoRA is parameter-efficient, it does not solve the problem of interference in generative retrieval. Because the document identifier distribution is a shared softmax, increasing the probability mass for new identifiers inherently reduces the mass for existing ones. Evidence from DSI++ research shows that this leads to forgetting, which requires techniques like rehearsal or sharpness-aware minimization to mitigate. A better approach is to use a delta index to achieve locality without forcing the optimizer to manage incremental updates.

Code playground (Python)