Evaluating LLMs for Production: Building a Golden Dataset That Actually Works
The first golden dataset we built for a client was written by the product team in an afternoon — forty clean, well-formed questions with obvious answers. It passed every time. Then we shipped, and real users asked messy, ambiguous, multi-part questions that looked nothing like our test set. The dataset told us the model was ready. Production told us otherwise within a day.
What actually works is pulling real queries from logs, staging environments, or support tickets — the messier the better — and having domain experts label the expected behavior, not just the expected answer. For a fintech client we built a 300-example set sourced entirely from six weeks of anonymized support transcripts, weighted toward the edge cases that had generated escalations, not the easy questions nobody complained about.
We also learned to separate "correctness" from "acceptable." Some queries have one right answer; many have a range of acceptable responses, and grading them pass/fail against a single reference answer produces noisy, misleading scores. We moved to rubric-based grading with an LLM-as-judge for the subjective categories, calibrated against human ratings on a 50-example subsample until judge and human agreement passed 90%.
The dataset isn't static. Every production incident that traces back to a model failure gets added as a new test case before the fix ships, which means the eval suite grows in exactly the direction that matters — the failure modes that actually happened, not the ones we imagined. Eighteen months in, one client's golden set has grown from 300 to over 1,200 examples this way.
The discipline that made the biggest difference wasn't the tooling, it was refusing to let anyone add "easy" examples just to keep the pass rate looking healthy. A golden dataset that's 95% softball questions gives you a 95% pass rate and no useful signal. We'd rather see 70% on a hard, representative set than 95% on one that flatters the model.
