← All questions
Retrieval & EmbeddingsHard· BM25· Relevance Feedback· RSJ Estimation

You have five hundred human relevance judgments across...

With 500 judgments across 50 queries, you have an average of 10 judgments per query. This allows you to estimate the S parameter in the BM25 formula. You should show the sensitivity of the weights numerically. As you collect more data, such as 50,000 judgments, RSJ estimation becomes more defensible, and the labels should be shifted toward training a dedicated reranker.

Code playground (Python)