How would you design a pairwise annotation process that...

Designing an annotation process that avoids length bias requires a multi-faceted approach. First, annotators should be presented with length-matched response pairs whenever possible to prevent them from defaulting to the longer answer. Second, add an explicit annotation dimension for conciseness, separate from overall quality, to force annotators to evaluate content quality independently of length. Third, run calibration passes where annotators compare known-correct short responses against plausible-but-longer hallucinated responses to train out the length heuristic. Finally, if using AI judges, compute preference scores as a function of the length difference across the evaluation set and apply a length-normalization correction before reporting win rates.