Mike HeapMike is an experienced Product Manager who focuses on all the “non-development” areas of My AskAI, from finance and customer success to product design, copywriting, testing and more.
Table of Contents
- AI hallucinations, in more depth
- How do AI hallucinations actually happen?
- What does a "good" AI hallucination rate look like?
- Common misconceptions about AI hallucinations
- Myth 1: "Hallucination" usually means the AI invented something out of thin air
- Myth 2: A smarter model will fix it
- Myth 3: "Zero hallucinations" is a meaningful vendor promise
- Myth 4: Hallucination is the same as a missing help-center article
- What AI hallucinations are NOT, and how they relate to adjacent terms
- How does My AskAI handle AI hallucinations?
- FAQs
Last Updated
Created time
May 26, 2026 10:27 AM
Title length (<60)
Author
Ecomm?
Image
Publish date
May 26, 2026
Slug
what-is-ai-hallucination
Featured
Featured
Type
Article
Ready to Publish
Ready to Publish
Vendors promise "zero hallucinations" but most can't show their work. The reality: in deployed customer-support AI, the majority of reported "hallucinations" aren't hallucinations at all (they're stale source docs the AI is faithfully grounding in). Here's the difference, why it matters, and the audit trail you should demand from any vendor.
An AI hallucination is when a generative AI model produces a confident, fluent, well-formed answer that is factually wrong: invented, misattributed, or unsupported by its source material.
The reason hallucinations are dangerous is the tone. The model writes a wrong answer in the same voice it writes a right one, so the customer reading the reply has no surface signal that anything's off.
That's what makes it different from a missing help-center article (where the AI should be saying "I don't know" instead), a typo (which a reader can correct around), or a refusal (where the AI explicitly declines to answer).
In our experience running an AI customer-support agent in production, most of what teams call a hallucination isn't one. They look at the help-center article or policy doc the AI was using and realize the source was already wrong, ambiguous, or out of date.
The model didn't fabricate; it faithfully grounded in a false truth. The fix for that pattern is completely different from the fix for the model-side one, and I'll come back to it in the misconceptions section.
If you're on this page you're probably evaluating a customer-service AI and trying to work out how often this is going to happen, why it happens, and what to demand from a vendor to know they actually prevent it. That's what the rest of this post is about.
AI hallucinations, in more depth
TL;DR: Hallucinations are a property of how transformer models work (the model predicts the next plausible token, with no objective on truth), but the failure mode is specific, measurable, and addressable by the layer you wrap around the model. We see this directly across customer rollouts.
Hallucinations are a property of how large language models work. The model is trained to predict the next plausible token given the tokens that came before. Truth doesn't enter the prediction objective.
When you ask it something it doesn't have grounded knowledge of, it produces the most plausible-sounding continuation, even when that continuation is fiction. The OpenAI paper Why Language Models Hallucinate (Kalai, Nachum, Vempala, Zhang) puts it more directly.
Language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty. A model that confidently guesses gets a higher score on most benchmarks than a model that admits it doesn't know.
There's a taxonomy worth knowing because vendors use it. Wikipedia's article on the topic splits hallucinations two ways:
The first split is intrinsic (the output contradicts the source material the model was given) versus extrinsic (the output can't be verified from the source). The second is closed-domain (the output contradicts the prompt) versus open-domain (the output doesn't contradict the prompt but invents something the model can't possibly know).
Here's the thing for a support-team buyer; hallucination is a measurable, addressable failure mode.
It's not random bad luck, and it's not unique to any one model. It clusters in predictable places: around the edges of training data, ambiguous prompts, and questions the AI's docs don't cover. Each of those places has a different fix.
How do AI hallucinations actually happen?
TL;DR: Five mechanism-side causes (training pressure, no uncertainty signal, no retrieval grounding, no refusal layer, decoding randomness), plus one source-side cause most explainers miss (and the one we see most often): the AI faithfully grounding in a wrong document.
The mechanism, in five steps, looks like this:
- Training pressure rewards fluent text. The model is rewarded for producing plausible, well-formed continuations. It is not rewarded, in the base training step, for refusing.
- The model has no native "I don't know" signal. There's no internal confidence score the model emits by default. It produces a probability distribution over next tokens, and that distribution isn't the same as an uncertainty score about facts.
- A question lands outside the training data. The user asks something the model hasn't seen. With no grounded answer to pull from, the next-token prediction fills the gap with the nearest-plausible continuation.
- There's no retrieval grounding to catch it. If the deployment doesn't use retrieval-augmented generation (RAG) to ground the answer in your actual documents, step 3's nearest-plausible guess goes out the door unchallenged.
- There's no refusal layer. Even when the model is grounded, the deployment might not enforce "if the retrieved context doesn't cover the question, refuse and escalate." Without that gate, a grounded model can still drift into invention on edge cases.
The decoding step adds randomness on top of all of this. MIT Sloan EdTech's guide on hallucinations names it directly.
Temperatures around 0.7-1.0 (used for creative work) produce more invention than temperatures around 0-0.3 (used for factual answers). A model that's grounded but running hot will hallucinate more than a model that's grounded and cold.
There's a parallel causal story the consensus AI explainers miss, and it matters more for customer-service AI than the model-mechanics version. The model can be perfectly grounded, perfectly calibrated, and still produce a wrong answer, because the source document it was grounded in was wrong.
I've seen all three across customer rollouts:
- A stale help-center article that says "30-day returns" when the policy moved to 14 days.
- A macro that was never updated after a brand rename.
- A docs page written before the product released a new tier.
The AI faithfully repeats the false truth. Here's the thing: AI is very good at grounding in truth and not making things up these days, but when you give it a false truth it doesn't know not to use it.
That class of failure has a different fix from the model-side class (you clean the source data; the model is doing its job).
Vendors differ on the mitigation stack they wrap around the model. Some lean heavily on RAG and grounding, some add an explicit refusal layer with thresholds for retrieval confidence, some show source citations on every answer.
We do something a little different (more on that further down). The general point: the causes of hallucination are the same across LLM-based products, but the mitigation stacks are not. That's what a buyer should be comparing.
What does a "good" AI hallucination rate look like?
TL;DR: Hallucination rate is meaningless without the refusal rate next to it. World-class (in our view) is under 1% on in-scope questions with over 95% refusal on out-of-scope. On the AA-Omniscience benchmark (May 2026): Claude Opus 4.7 sits at 36%, Gemini 3.1 Pro at 50%, GPT-5.5 at 86%.
There's no universal benchmark, but a few public numbers from May 2026 help calibrate the conversation. The benchmark to know is AA-Omniscience from Artificial Analysis, which is designed specifically to surface hallucination.
AA-Omniscience counts a fabricated answer as a hallucination and a refusal as a non-hallucination. So the percentages it reports aren't "% of every response is wrong"; they're "% of attempted answers that were fabricated when the model couldn't be sure". That distinction matters a lot (take any vendor's "hallucination rate" with a grain of salt until you know which version they mean).
The May 2026 frontier-model numbers on AA-Omniscience:
Model | Released | Hallucination rate | Omniscience Index |
Claude Opus 4.7 | April 16 2026 | 36% (down from Opus 4.6's 61%) | 26 (2nd highest) |
Gemini 3.1 Pro | February 2026 | 50% (down from Gemini 3 Pro's 88%) | 33 (highest) |
GPT-5.5 | April 2026 | 86% | — |
GPT-5.5 Pro (extended thinking on) | April 2026 | 4.2% | — |
Grok 4.20 (Reasoning) | April 2026 | 17% | — |
The numbers tell three different stories.
Anthropic's calibration choice on Opus 4.7 is to refuse rather than guess. The 36% rate is on attempted answers, and the model deliberately reports an error when it doesn't have enough grounded context (Opus 4.7's long-context retrieval dropped to 32.2% versus Opus 4.6's 78.3% specifically because of that calibration shift).
Google's calibration on Gemini 3.1 Pro is to attempt everything. That gives it the highest Omniscience Index (it knows more when it's correct), but pairs the high knowledge with a 50% fabrication rate on uncertain questions.
OpenAI's default on GPT-5.5 keeps the older accuracy-over-calibration tradeoff. It's 86% on uncertain questions by default, but the extended-thinking variant (GPT-5.5 Pro) drops to 4.2% when the user explicitly enables reasoning.
For a customer-support AI buyer, the takeaway isn't the absolute number (we tell every buyer this on demo calls). It's the strategy the vendor's deployment is taking.
The two questions I'd want answered before I signed anything:
- What's the refusal rate, alongside the hallucination rate? A vendor claiming 1% hallucination who only refuses 5% of questions isn't the same product as a vendor claiming 1% hallucination who refuses 50% of questions. The second is calibrated.
- Is grounded retrieval and refusal on by default, or does the user have to toggle it? GPT-5.5 sits at 86% by default, 4.2% with reasoning on. The default behavior is what reaches your customers.
A banded frame for buyer conversations, conditioning hallucination rate on refusal rate:
Tier | Look for |
World-class (refusal-grounded) | <1% on in-scope questions; refuses >95% of out-of-scope |
Solid | <5% on in-scope; refuses >80% of out-of-scope |
Average | 5-15% on in-scope; refuses 50-80% of out-of-scope |
Needs work | >15% on in-scope; refuses <50% of out-of-scope |
Regulated industries (finance, health, legal, airline, where the Air Canada and Mata v. Avianca precedents argue for it) should be one tier stricter on every floor.
Common misconceptions about AI hallucinations
TL;DR: Four myths drive most bad buying decisions (we hear all of them on demo calls). Most reported "hallucinations" turn out to be stale docs. A smarter model rarely fixes the underlying calibration problem. "Zero hallucinations" is a marketing claim with no enforceable guarantee. And hallucination is a distinct failure mode from a missing help-center article.
A handful of myths drive most bad buying decisions in this category. We see them on demo calls every week.
Myth 1: "Hallucination" usually means the AI invented something out of thin air
This is where a lot of teams waste their first three months chasing the wrong fix.
Most of what people think of as hallucinations aren't actually hallucinations. They look at a bad AI reply and assume the model made it up, until they look at the information they gave it and realize their docs or help center was out of date, ambiguous, or just wrong.
AI is very good at grounding in truth and not making things up these days. But when you give it a false truth it doesn't know not to use it.
Real (model-side) hallucinations do exist. The AA-Omniscience table above is real, and the Mata v. Avianca case is real.
But the most common buyer-reported "hallucination" we see in deployed support AI is the source-data version: the AI faithfully repeated something a stale help-center article said. The fix for the model-side version is a better grounding and refusal stack.
The fix for the source-data version is cleaning the source data. They're different jobs (and they need different tools).
Myth 2: A smarter model will fix it
It won't, at least not on its own. Look at the AA-Omniscience table above again.
Anthropic's Opus 4.7 cut Opus 4.6's hallucination rate from 61% to 36%, a real, durable generational improvement. But GPT-5.5 in the same generation hallucinates 86% of attempted answers, the most extreme accuracy-vs-calibration gap yet observed.
Three labs in the same year produced three very different hallucination behaviors from broadly similar underlying technology (which on its own should tell you the model isn't the lever). The OpenAI paper Why Language Models Hallucinate spells out why: hallucinations are a property of how models are evaluated and rewarded as much as how they're trained.
The fix sits in the layer around the model: retrieval grounding, refusal thresholds, citation surfaces. From our work with each generation, picking the next version of the same model rarely changes the calibration story.
Myth 3: "Zero hallucinations" is a meaningful vendor promise
It isn't, and it almost always means something narrower than the claim suggests. CoSupport AI's pricing page, for example, advertises "AI support agents that don't hallucinate, live in 10 mins".
Alhena AI made similar claims in older marketing copy.
In our experience auditing vendor pages (see our Alhena AI breakdown and CoSupport AI breakdown for two worked examples), the "zero hallucinations" promise is usually one of two things:
- It's either a refusal-conditioned measurement (the model refuses on out-of-scope, and the vendor measures hallucination only on the subset where it answered), or
- A knowledge-base-conditioned measurement (the vendor counts a wrong answer as the customer's docs being wrong, and excludes it from the hallucination metric).
The right buyer question isn't "do you promise zero hallucinations?". It's "on any one of my last 100 tickets, show me exactly which knowledge source you used and the reasoning chain you took."
A vendor who can do that is one whose claim you can verify. A vendor who can't is selling a marketing line.
Myth 4: Hallucination is the same as a missing help-center article
It's not. A missing entry is what the docs don't cover (the question the AI has no grounded answer for). The correct response to that absence is a refusal and an escalation to a human.
Hallucination is what happens when that refusal fails: when the missing entry triggers an invention instead of an "I don't know". The two failure modes are connected but they're not the same, and they have different fixes.
The first is closed by Self-Learning routines (the AI surfaces the gap, the team writes the article). The second is closed by the refusal layer.
What AI hallucinations are NOT, and how they relate to adjacent terms
TL;DR: Hallucination is not the same as a coverage gap, a refusal, bias, misinformation, or RAG. Each is a distinct failure mode (or, in RAG's case, a mitigation) with a different fix. The taxonomy is what tells you which job to do first.
There are a handful of terms that get used interchangeably with "hallucination" and shouldn't be (we hear this confusion on most demo calls). Each is a distinct failure mode with its own fix.
Term | What it is | How it differs from hallucination |
Coverage gap (missing help-center article) | The knowledge base doesn't cover the answer | A coverage gap is the cause that should produce a refusal. Hallucination is what happens when refusal fails and the AI invents an answer instead. |
Refusal | The model declines to answer ("I can't answer that, let me hand you to a human") | The opposite of hallucination. Refusal is the correct fallback when grounding fails; hallucination is the failure mode when refusal doesn't fire. |
Bias | Skewed pattern in model output reflecting training data | Bias is wrong-on-average across a population. Hallucination is wrong-on-specifics for one user. |
Misinformation | False content the user is exposed to | Hallucination is one source of misinformation, but not the only one. Stale docs, malicious prompts, and out-of-date macros can all produce misinformation without any hallucination. |
RAG (retrieval-augmented generation) | Architecture that grounds answers in retrieved documents | RAG is the most common hallucination mitigation, not a definition of the failure mode itself. |
The reason these distinctions matter is operational. A team that confuses hallucination with a missing help-center entry will rebuild their RAG stack when they should have written a docs article.
A team that confuses hallucination with bias will demand a different model when they should have written a refusal rule. The taxonomy is what tells you which job to do first.
How does My AskAI handle AI hallucinations?
TL;DR: Grounding-first stack (RAG over your knowledge base + refusal-by-default) plus Inspect & Logs (open any conversation in your dashboard and ask "why did you give this answer?" to see the exact source documents and reasoning chain). RecruitCRM went from 35% to 68% AI resolution rate using exactly this audit loop.
Our stack has four parts. The first three are what every serious vendor should ship; the fourth is the one we lean on more than anything else.
- Grounded retrieval over your help center, public website, PDFs, SOPs, historical tickets, and (where you connect them) Google Drive, Notion, Confluence, SharePoint, OneDrive, Dropbox, Salesforce, and Shopify. Where it's genuinely helpful to the end customer, our AI links to a relevant help-center article inside its answer, but we don't surface a separate "sources used" footer by default (it clutters the conversation). The full source list lives in Inspect (next).
- Refusal-and-escalate when the retrieval doesn't return enough confidence on a question. Table stakes for a serious vendor.
- Self-Learning, which closes the source-data class of failure over time. When a human agent corrects an AI reply on a handed-over ticket, Self-Learning surfaces the delta and drafts a new knowledge article so the next time the same question comes in, the AI grounds in updated information.
- Inspect & Logs: the per-conversation audit feature. This is the one we lean on. Open any conversation in your account and ask the AI directly: why did you give this answer? and where did you get that information from? You see the specific knowledge sources it pulled from along with its reasoning chain.
We built Inspect because I got tired of hallucination conversations being unfalsifiable. When a customer complained that the AI had "made something up", there was usually a real source document the AI had pulled from. Once we could show that document to the team, the conversation moved from "the AI is broken" to "this help-center article needs updating" in about thirty seconds.
Since we launched Inspect, I can't remember the last time we had a customer complain about a true hallucination. The model still invents things occasionally, of course. What the audit layer changes is the conversation; it turns invisible model-side failures into auditable source-data ones.
Combined with Internal Notes mode (where the AI drafts every reply as an internal note before any of it touches a customer), it gives a team a way to validate behavior with zero customer-facing risk before going live.
For a named customer example: RecruitCRM, the all-in-one SaaS platform for recruitment agencies, runs My AskAI inside Intercom and gets 68% AI resolution rate (up from roughly 35% at go-live). They save around 62 hours per month at 75% AI CSAT across roughly 5,700 tickets resolved by the AI in the first year.
The detail that matters for this post: the resolution-rate climb from 35% to 68% came from a disciplined weekly QA review process. The team opened conversations the AI didn't handle well, looked at exactly which source the AI used, fixed stale sources, and added custom answers where the docs didn't cover the question at all. The lever was the audit trail and the data; the model never changed.
If you want to see Inspect on your own tickets, you can start a free trial and have it running inside Zendesk, Intercom, Freshdesk, Gorgias or HubSpot in about ten minutes.
FAQs
What is an AI hallucination?
An AI hallucination is when a generative AI model produces a confident, fluent answer that is factually wrong: invented, misattributed, or unsupported by its source material.
The danger is the tone (the model writes a wrong answer in the same voice it writes a right one), so the reader has no surface signal that anything's off.
What does AI hallucination mean?
In our day-to-day on demo calls, "hallucination" is shorthand for two related things. One is a fabricated answer the AI invented because it had nothing grounded to pull from.
The other is a faithfully grounded answer that happened to come from a wrong source document. Both shapes are commonly called hallucinations. The technical literature usually means the first; buyers usually mean the second.
What's the difference between an AI hallucination and a missing help-center article?
A missing help-center article is what the AI's docs don't cover (a question with no grounded answer available). The correct response to that absence is a refusal and an escalation.
A hallucination is what happens when that refusal fails and the AI invents an answer instead of saying "I don't know". Same underlying cause (no grounded source), different outcome (refusal versus invention).
What's the difference between an AI hallucination and a refusal?
A refusal is the AI declining to answer when it doesn't have enough confidence in the available information: "I can't help with that specifically, let me hand you over to a human."
A hallucination is the opposite (the AI confidently answering when it shouldn't have). Refusal is the correct fallback behavior; hallucination is the failure mode that fires when refusal doesn't.
What causes AI hallucinations?
Five things, in roughly this order (and we see all of them across rollouts). Training rewards plausible text rather than truthful text. The model has no native uncertainty signal.
The question lands outside its training data. There's no retrieval grounding in your actual documents. There's no refusal layer to enforce "if not grounded, don't answer."
There's also a sixth cause people miss: the grounding worked, but the source document was wrong, so the AI faithfully repeated a false truth.
What are some examples of AI hallucinations?
The well-known ones: Mata v. Avianca (2023), where two US attorneys submitted a ChatGPT-drafted brief citing six fabricated court cases and were sanctioned $5,000 under Rule 11; Moffatt v. Air Canada (2024), where Air Canada's chatbot invented a bereavement-fare policy and the BC Civil Resolution Tribunal ordered C$812 in damages; Meta's Galactica (2022), which produced fictitious academic papers and was withdrawn within days.
In customer support specifically, the most common shape isn't a famous lawsuit. It's an AI walking a user through a setup that no longer matches the product because the help-center article was last updated 18 months ago.
How do I prevent AI hallucinations in a customer-support AI?
Three things, in priority order. First, ground every answer in your actual knowledge base via RAG rather than relying on the model's training data alone.
Second, enforce refusal-by-default: if the retrieval doesn't return enough confidence, the AI should escalate to a human instead of guessing. Third, run a per-conversation audit feature (like our Inspect & Logs) so that any time something looks wrong, you can open the conversation and see exactly which source the AI used.
The third one is the operational one. It's how you find out whether your "hallucinations" are model-side failures or stale-docs failures.
What is a "good" AI hallucination rate?
There isn't a single answer, because the number is meaningless without the refusal rate alongside it. In our experience, a vendor claiming 1% hallucination who refuses 5% of out-of-scope questions is not the same product as a vendor claiming 1% hallucination who refuses 80% of out-of-scope.
The second is calibrated. As a rough buyer-side band: world-class is under 1% in-scope with over 95% refusal on out-of-scope, solid is under 5% with over 80% refusal, average is 5-15% with 50-80% refusal, and under either floor you shouldn't deploy it customer-facing.
How can I detect AI hallucinations in my deployed support AI?
Two complementary methods (we use both internally). The first is sampling: pick 100 recent conversations at random, open each, and check the AI's answer against the source. This is slow but high-confidence.
The second is a per-conversation audit feature where you can ask the AI to explain its reasoning and show its sources for any specific conversation. The second method is what makes routine detection feasible at scale; the first is what you use to validate the second.
Are AI hallucinations getting worse with newer models?
Mixed. On the AA-Omniscience benchmark in May 2026, Claude Opus 4.7 cut its predecessor's hallucination rate from 61% to 36% (a clear generational improvement driven by Anthropic's calibration-first strategy).
Gemini 3.1 Pro cut Gemini 3 Pro's 88% to 50%. But GPT-5.5 in the same generation sits at 86% on uncertain questions by default.
So the answer's vendor-specific rather than generation-specific (and "newer equals better" isn't reliable).
Does RAG eliminate AI hallucinations?
No, but it reduces them substantially when implemented properly. RAG (retrieval-augmented generation) grounds the model's answer in retrieved documents rather than its training data, which closes the model-side hallucination path.
What RAG doesn't fix is the source-data class: if your retrieved document is wrong, the model will faithfully repeat the wrong content. The fix for that is auditing the sources (the retrieval is doing its job). A serious vendor pairs RAG with refusal-by-default and a per-conversation audit trail, so both classes get covered.
What should I ask an AI customer-service vendor about hallucinations?
Five questions, in this order. (1) "On any one of my last 100 tickets, can you show me exactly which knowledge source the AI used and the reasoning chain it took?". If they can't, they don't have an audit layer.
(2) "What's the refusal rate alongside the hallucination rate?". The second number is meaningless without the first. (3) "When the retrieval doesn't return enough confidence, what does the AI do by default?". The right answer is "refuse and escalate."
(4) "How do you handle a wrong source document, when the docs are out of date or ambiguous?". They should be able to describe their Self-Learning or knowledge-update loop.
(5) "Show me the most recent product page or pricing page where you make a 'zero hallucinations' claim, and walk me through the mechanism behind it.". Vendors who can't decode their own claim are selling marketing.
