We’re seeing a gap between our benchmark scores and use...

The discrepancy between benchmark scores and production satisfaction is typically caused by the difference between quiz-style benchmarks and real-world prompts. Benchmarks rely on clean ground truth, whereas production prompts are often context-dependent and underspecified. To diagnose the issue, sample failing production sessions and analyze them for structural properties—such as long context, cross-file references, or ambiguous intent—that are absent from your current benchmarks. Once identified, build a targeted ask-style evaluation that specifically captures these properties.