Should you trust an attribution judge with 95% accuracy...

High accuracy can be misleading if the test set is imbalanced. A judge that defaults to ‘attributable’ for every claim might achieve 95% accuracy while failing to identify any ‘not attributable’ cases. Macro-F1 is a more reliable metric for evaluating performance across all classes, ensuring the judge is actually capable of identifying the errors you are trying to catch.