A domain assistant that answers ninety-five questions correctly and invents the ninety-sixth is more dangerous than one that admits ignorance, because users extrapolate trust from the confident wrong answer. Hallucination risk is not a vague worry you handle with a disclaimer — it is a measurable property of your system, and you can put numbers on it long before your customers do.
Why domain assistants hallucinate
Language models generate the most plausible continuation of a prompt, not the most verified one. In a narrow domain your sources cover most questions well, but the edges — rare products, outdated pricing, policy exceptions, unsupported configurations — are exactly where the model fills gaps with fluent invention. Three failure modes dominate in practice:
- Ungrounded specifics: numbers, names, dates or steps that appear nowhere in your documentation or knowledge base.
- Stale general knowledge: facts carried over from pre-training that were true once but now conflict with your current sources.
- Blended answers: real details stitched together with plausible fabrications so smoothly that only a domain expert notices the seam.
Build a test set from real questions
An evaluation is only as honest as its questions. Collect between fifty and two hundred prompts that reflect what people actually ask, not what you hope they ask:
- Pull real questions from support tickets, search logs and chat transcripts — verbatim, including typos and vague phrasing.
- Add time-sensitive questions about pricing, availability and policy, since these drift and break first.
- Include questions your sources genuinely cannot answer; the correct output is a refusal, and that behaviour must be scored too.
- Ask a domain expert to flag anything where a wrong answer would cost money, safety or legal exposure — those need the strictest bar.
Score answers claim by claim
Do not grade a response as a single pass or fail. Split each answer into individual claims and check every claim against a source: does it appear in, or follow logically from, the retrieved context? Three metrics fall out of this, and they are worth tracking separately:
- Grounded-answer rate — the share of answers where every factual claim traces to a source.
- Fabrication rate — the share of answers containing at least one unsupported claim.
- Refusal correctness — how often the assistant properly declines questions that have no answer in your sources.
You can run claim checks with a strong evaluator model, but spot-check its judgements by hand. Evaluator models are themselves fallible, and an unexamined scorecard breeds false confidence.
Test refusal behaviour explicitly
The cheapest hallucination to fix is the one the assistant never produces. For every unanswerable question in your test set, record whether the model refuses, hedges or invents. A healthy domain assistant declines clearly, states what it cannot answer, and redirects to a human or a document — it never guesses quietly. If refusal correctness is low, fix it with instructions and retrieval constraints before you touch training data.
Probe the edges before users do
Once the core set passes, attack it:
- Paraphrase the same question five different ways; grounded answers should stay stable in content.
- Add irrelevant context or red herrings and see whether the assistant gets pulled off-source.
- Ask compound questions that mix answerable and unanswerable parts — the ideal reply answers one part and declines the other.
- Probe near-miss entities: similar product names, old plan tiers, discontinued features.
Keep watching after launch
A test set is a snapshot; production is a movie. Sample live conversations weekly and run them through the same claim-level scoring, route low-confidence answers to human review, and watch indirect signals such as thumbs-down clicks, repeated rephrasing and escalations. When the grounded-answer rate drifts down after a model update, a prompt change or a document refresh, treat it as a regression and roll back or patch — exactly as you would with any other quality metric.
Constrain first, retrain second
When the numbers disappoint, the fastest fix is almost always architectural rather than statistical. Ground the assistant with retrieval so it answers only from supplied context, require citations, and cap generation length. Fine-tuning still earns its place — it improves tone, format and refusal style — but it does not guarantee factuality. Evaluate after every change, keep the scoreboard visible, and hallucination risk becomes an engineering metric you manage instead of a fear you live with.
