Our assistant is wrong on rare SKUs. The ML lead wants...

Using continued pre-training to inject factual knowledge about 200,000 product documents into model weights suffers from poor scaling efficiency. The exchange rate for memorization during pre-training requires approximately two decades of parameter scale per decade of documents. Spreading 200,000 documents across a massive catalog means the average document frequency $d$ per unique fact remains extremely low ($d \approx 1$), far below the threshold required for reliable parametric recall.

In addition to high compute costs, continued pre-training fails to address fundamental limitations of parametric memory:

1. Co-occurrence Bias: Additional pre-training epochs tend to amplify existing co-occurrence biases rather than fix precise factual lookups for tail items.
2. Static Knowledge: Knowledge remains frozen at the time of the training run, requiring additional training passes whenever product specs or inventory change.

Applying parameter-efficient methods like LoRA reduces the compute bill but does not alter the fundamental accuracy surface for long-tail factual retrieval. Placing the 200,000 product documents into a retrieval index (RAG) is significantly cheaper, allows real-time updates, and provides far higher recall on rare SKUs.