Most teams judge an AI assistant the way they judge a new hire in their first week: a quick gut feeling after a handful of interactions. The problem is that a gut feeling does not survive contact with real users. Ten friendly questions in a demo tell you almost nothing about how the assistant will handle the four hundred strange, ambiguous, genuinely hard questions it faces in month one. If you want to know whether your assistant is any good, you have to measure it — with the same seriousness you would apply to any other production system. And it has to be measured on your questions, with your definition of a good answer, not on a public benchmark that has nothing to do with your business.
Start with a golden set of real questions
The foundation of every evaluation is a fixed set of questions and reference answers, often called a golden set. Mine it from the places your real questions already live: support tickets, sales threads, chat transcripts and search logs. Collect the questions that actually occur, not the ones marketing would write. A useful starting mix:
- The 60 percent that come up every week — pricing, shipping, refunds, how something works.
- The 20 percent that are occasional but important — edge cases, policy exceptions, rare configurations.
- The 20 percent that should be refused or escalated — requests for legal advice, account changes, anything where a wrong answer costs money.
Write a reference answer for each question, even a rough one. Fifty well-chosen questions tell you more than five thousand synthetic ones.
Metrics that actually matter
Vanity scores are everywhere in AI. Ignore most of them and track a small number of numbers you can act on:
- Answer accuracy — the share of answers a reviewer would sign off on as correct and complete.
- Groundedness — whether claims trace back to your documentation or data, which separates a trustworthy assistant from a merely confident one.
- Refusal quality — how often the assistant correctly says it does not know or hands off to a human, and how often it refuses when it should not.
- Task completion — did the user actually get their problem solved, measured by follow-up questions, ticket closure or a short post-chat survey.
- Latency and cost per answer — because a brilliant assistant nobody waits for is still a failure.
Score answers the way a reviewer would
Automatic metrics struggle with open-ended answers, so borrow a technique from human evaluation: grade against a rubric. For each answer, ask three yes-or-no questions. Is it factually correct? Is it grounded in the source material? Would you send it to a customer unchanged? Any no makes the answer a fail. Two or three reviewers scoring the same sample, with their agreement checked, keeps the standard honest.
You can also use a strong model as a first-pass grader: show it the question, the answer and the reference, and ask for a verdict with reasons. Model judges are consistent and cheap, but audit them — sample their judgments against human scores every few weeks and recalibrate when they drift. Reserve human review for the borderline cases the judge flags as uncertain.
Watch the production signals
Offline scores are the rehearsal; production is the performance. The good news is that the most useful signals cost almost nothing to collect:
- Escalation and retry rates — how often users abandon the assistant for a human channel or rephrase the same question.
- Explicit feedback — thumbs up and down, weighted toward what unhappy users do rather than what happy ones click.
- Silent failures — answers that were confidently wrong and never escalated; find these by reviewing a random sample of transcripts weekly.
- Question clusters — new topics in the logs are next month's documentation gaps.
Re-run the whole thing after every change
An evaluation set is only valuable if it is run consistently. Every prompt tweak, knowledge base update or model upgrade should trigger the same harness, the same questions and the same rubric, with results stored so runs can be compared. Keep the history; trends matter more than single runs. A change that lifts accuracy two points but quietly doubles refusals is not an improvement. This regression habit — more than any single score — is what keeps an assistant good over time, because quality decays silently when nobody is looking for it.
None of this requires a research team. A spreadsheet of a hundred real questions, a one-page rubric, a weekly transcript review and the discipline to re-test after every change will put you ahead of most teams shipping AI assistants today. Good is not a feeling you had during the demo — it is a number you can defend.
