← All questions
Retrieval & EmbeddingsMedium· Data Augmentation· Dense Retrieval· InfoNCE

You have 8 million internal documents and zero labeled...

Since negatives can be sampled, the core issue is the lack of positive pairs. You can generate these by creating independent crops of your existing documents. After training with InfoNCE, it is critical to evaluate the model against a baseline like BM25 using a small, hand-curated set of queries. This approach is superior to simply fine-tuning an existing checkpoint like MS MARCO, which often performs poorly out-of-domain.

Code playground (Python)