Can we drop the generator from 70B to 8B parameters whe...

When scaling a datastore by a factor of 100, the generator’s job becomes significantly more difficult. The top-k window will contain more noise, including near-duplicate and contradictory information that the model must synthesize. Reducing the model size from 70B to 8B parameters simultaneously reduces the model’s capacity to handle this increased complexity. You should measure the impact on performance before attempting such a swap.