← All questions
EvaluationHard· Inference· Scaling· Accuracy

Your grounded QA accuracy is 60% and the product wants...

Mathematical analysis shows that voting-based methods like self-consistency have diminishing returns. If the base accuracy (p) is 60%, increasing the number of samples does not scale linearly toward 75%. A large gap in accuracy suggests that the underlying probability of the model selecting the correct answer is low, likely due to poor retrieval. Investing in a reranker to ensure the supporting passage is consistently in the context is a more effective use of compute, as it raises the base p, which then multiplies the effectiveness of any downstream reasoning scaffold.

Code playground (Python)