A team reports a 5-point win rate improvement on Alpaca...

Before trusting a win rate improvement on AlpacaEval, you must verify three things. First, check whether the improvement survives length correction; a win rate that disappears after length correction is often measuring verbosity rather than quality. Second, check if the same improvement appears on Chatbot Arena or through human evaluation on your actual task distribution. Third, verify that the baseline was evaluated under identical decoding parameters, including temperature, top-p, and maximum tokens.