A colleague wants to scale the retrieval corpus from Wi...

At a 522x increase in corpus size, the rebuild cost becomes roughly 565x the trainer’s compute per window, effectively turning the task into an embedding job rather than language model pre-training. To handle this, you must choose between two compromises: freeze the document encoder to eliminate the retriever gradient, or maintain a trainable retriever over a small hot index while routing the cold tail through a frozen index, which sacrifices a single consistent scoring function across the entire corpus.