Your document-first decoder reranker profiles at 0.0155...

The discrepancy between theoretical FLOPs and actual latency is often due to execution overhead. Eager execution pays a memory round trip at every kernel boundary, while variable input shapes prevent effective graph compilation. Simply adding more GPUs or using a smaller model does not address the bottleneck caused by compilation retracing or eager execution overhead.