How would you build an ask-style eval for a general-pur...

For tasks without a single correct answer, you should layer three distinct signals. First, use LLM-as-judge at scale for triage, noting that these are fast and cheap but often biased toward verbosity. Second, audit a random 5% slice of the data with human raters to report judge calibration; if the LLM judge agrees with humans 80% of the time, you have a baseline for trust. Finally, track behavioral proxies such as acceptance rates, task completion, and return rates as a continuous production signal. A score without calibration metadata is merely an anecdote, not a true evaluation.