Your pass@1 improved by 4 points on HumanEval, but user...

The gap exists because HumanEval problems are short, self-contained, and have canonical test suites, whereas production code generation tasks involve existing codebases with implicit context and rarely have test suites. A 4-point gain on HumanEval may simply reflect better performance on easy function-completion problems that are already well-handled in production. To determine if the gain is real, evaluate the model on a held-out sample of real user sessions using metrics like accept rate or user edit distance. If the gain does not translate, the issue is in the evaluation distribution, and you should redesign the benchmark.