Mike HeapMike is an experienced Product Manager who focuses on all the “non-development” areas of My AskAI, from finance and customer success to product design, copywriting, testing and more.
Table of Contents
- How AI customer service actually breaks: the three stages
- Mistake 1: Pointing the AI at thin or stale knowledge
- Mistake 2: Choosing the tool on hype instead of fit (and a bill you can't forecast)
- Mistake 3: Shipping a scripted chatbot and calling it an "AI agent"
- Mistake 4: Deploying with no guardrails against confident wrong answers
- Mistake 5: Hiding the human (no clean escalation path)
- Mistake 6: Measuring deflection instead of resolution
- Mistake 7: Treating it as set-and-forget (expecting it to improve without you)
- What this looks like in real rollouts
- What to do this week
- When these aren't mistakes
- The takeaway
- FAQs
Created time
Jun 1, 2026 01:58 PM
Title length (<60)
Author
Ecomm?
Image
Publish date
Jun 1, 2026
Slug
common-ai-customer-service-mistakes
Featured
Type
Article
Ready to Publish
Ready to Publish
Most AI customer service rollouts fail the same seven ways, and almost all of them trace back to one root error: treating the AI as a switch you flip instead of a system you operate.
Let's be honest, day one is the worst your AI support agent will ever be. The most common mistake teams make is treating day one like it's the best. They switch it on, walk away, and then wonder why it stalled.
Most companies use AI somewhere in support now, and we're one of the tools they reach for. And yet 64% of customers say they'd rather companies didn't.
Read the Gartner detail, though, and it's not that people hate automation. Their top worry is that it'll get harder to reach a person, then that it'll give wrong answers.
So customers don't reject AI support. They reject badly operated AI support.
That's the pattern I keep seeing. A rollout half-works in week one, the team decides half-working is the ceiling, nobody feeds it, and it freezes there at its worst.
These failures are predictable, and they're rarely the model's fault. The same handful of mistakes shows up again and again, and they cluster into the three places an AI rollout tends to break. I'll map those out next.
We've shipped AI agents inside Intercom, Zendesk, Freshdesk, Gorgias and HubSpot for 200+ ecommerce and SaaS teams, from solo founders to ops handling 100,000+ tickets a month. Our agents have resolved over a million tickets, at a rolling resolution rate above 72%. I've watched the teams who put the work in double their resolution, and the ones who set-and-forgot stay exactly where they started.
Here are the seven mistakes, and the fix for each.
How AI customer service actually breaks: the three stages
TL;DR: AI support breaks in three places: what you aim it at, how you let it answer, and how you keep score. Each of the seven mistakes below lives in one of them.
Almost every failure below is a symptom of one root error. Teams treat AI support as a thing you switch on, rather than a system you operate. It helps to know which part of the system a given mistake lives in, so that's how we've grouped them.
Stage | The question it answers | Mistakes |
1. What you aim it at | Is the AI pointed at good, current info and the right tool? | 1 and 2 |
2. How you let it answer | Does it answer well, safely, and hand off cleanly? | 3 to 5 |
3. How you keep score | Are you measuring the right thing, and improving it? | 6 and 7 |
Mistake 1: Pointing the AI at thin or stale knowledge
TL;DR: An AI agent is only as good as the knowledge you feed it. Most "bad answers" are stale or thin docs, so write your knowledge down and keep it current.
An AI agent is only ever as good as what you give it. Most of the "the AI gave a bad answer" complaints I see aren't model failures at all. They're knowledge failures, where the AI was pointed at docs that were thin, out of date, or ambiguous, and it answered confidently from the wrong place.
It's still garbage in, garbage out. If the thing a customer needs isn't written down somewhere the AI can read, the AI has nothing to learn from.
This is also where most so-called hallucinations actually come from, in our experience. Modern models are good at grounding in the truth you give them, but bad at spotting when that truth is wrong. Hand the AI an out-of-date returns policy and it'll quote the out-of-date returns policy, confidently.
The fix is unglamorous, but luckily it's simple. Write your knowledge down and keep it current, because the fastest gains we see come from exactly that.
Edel Optics, the European eyewear retailer, sat at 20-30% resolution on help-center FAQs alone. Then they connected live order, delivery and tracking data, and resolution jumped to around 79% almost overnight, a leap of roughly 50 points. Same model, better inputs.
Pointing the AI at your real material also kills the "we're not ready" excuse. Historic ticket training lets a company with no help center at all stand up an agent from its past resolved tickets, and Self-Learning then drafts fresh knowledge from the replies your human agents send after a handover. The knowledge base maintains itself, but only once you've given it something to build on.
Mistake 2: Choosing the tool on hype instead of fit (and a bill you can't forecast)
TL;DR: Pick your tool on fit rather than hype, and choose a price meter you can forecast. Per-resolution bills can double or triple as your AI improves.
The most expensive mistake happens before launch. You pick a tool on brand, hype, and a headline resolution number, then find a price meter you can't forecast.
The hardest part of choosing an AI support tool, in my experience, is working out what it'll actually cost you without a sales call. Some vendors price per reply, but you don't know how many replies you'll send next month. Some price per resolution, but you've no idea what resolution rate you'll get on day one, let alone in a year.
Others bill in credits that aren't obvious, or even tokens (which only make sense if you live in this stuff). You tend to find out the real number once you're already inside the product.
Per-resolution pricing has a nasty trap. If you start at 20-30% resolution and climb to 70-80% over a year (which is the whole point), your bill on a per-resolution model can double or triple, because you pay more precisely as the AI gets better.
And most of what drives that climb is your own work, like better knowledge, connected APIs, and tuned guidance. Outcome-based pricing charges you for the upside your own effort created.
We've watched buyers do this maths and walk away. Kriptomat, the EU crypto exchange, rejected Intercom Fin at $0.99 per resolution as uneconomical at their volumes. Freecash tested Fin first and reached the same conclusion, and both went with a per-ticket model instead.
That's our model too, at $0.10 per ticket. We bill per ticket rather than per resolution on purpose, since the bill is then counted in tickets and most teams can predict those.
Whatever you end up buying, the fix is the same. Pick a meter you can forecast your own usage of before you sign.
Mistake 3: Shipping a scripted chatbot and calling it an "AI agent"
TL;DR: A scripted decision-tree bot is not an AI agent. Demand one that understands a question it has never seen and can take an action on your systems.
There's a real difference between a chatbot and an AI agent, and I've watched teams automate frustration by getting it wrong. A decision-tree bot that matches keywords and marches customers through a fixed flow is not an AI agent. The second a customer asks something the tree doesn't cover, it dead-ends.
The line I'd draw is between merely retrieving knowledge and actually doing a job on the customer's behalf. A real agent understands a question it's never seen, finds the answer, and where needed takes the action (looking up an order, processing the cancellation, updating the account) by calling your systems. In our product that's the gap between answering and Tasks, which are natural-language, multi-step procedures that replace decision trees entirely.
The flip side is not over-building. You don't need a workflow for everything that takes more than one reply.
A workflow earns its place only when resolving the ticket genuinely needs several steps and a call out to another system. For everything else, good knowledge plus a model that understands the question is the agent (no workflow required, and we'd steer you away from one).
At selection time, don't take "AI" on the label at face value. Ask to watch it handle a question that's in no script, and ask whether it can take an action or only answer.
Mistake 4: Deploying with no guardrails against confident wrong answers
TL;DR: The real risk is a confidently wrong answer you never see. Use guidance, test in internal-notes mode, and keep an audit trail so your team can check why the AI answered.
The danger with AI support is the confidently wrong answer you never see. An "I'm not sure" reply is the easy case. Without controls and a way to trace what happened, a wrong answer looks exactly like a right one until a customer complains.
Three things keep this in check. Guidance (natural-language rules for tone, terminology and when to hand off) keeps the AI inside the lines you set. Internal-notes mode lets you run the AI silently next to your existing setup, drafting replies as private notes your team reviews, so you check quality before a single customer sees an AI reply.
The third is an audit trail. In our dashboard the team can open any conversation and ask the AI "why did you give this answer?" or "where did you get that information?" and see exactly what knowledge it used (it's a team view that the customer never sees).
Since we built that, I honestly can't remember the last time someone hit a true hallucination. The handful that looked like one turned out to be the stale-document problem from Mistake 1, and the trail proved it.
One caveat worth saying plainly. Even Self-Learning needs a human in the loop, because plenty of tools claim self-learning and don't really have it, and even where it's real, you should review what's being added (both transcription and the AI's read of a human agent's reply leave room for error).
The fix here is never go live wide open. Test in notes mode, set your guidance, and keep the audit trail somewhere your team can actually reach it (ours sits one click from any ticket).
Mistake 5: Hiding the human (no clean escalation path)
TL;DR: Make it easy for customers to reach a human. Hiding the human inflates your resolution number and frustrates the people you most need to keep.
Remember the top reason customers are wary of AI support? They're scared it'll be harder to reach a person (we saw that worry top the Gartner list earlier).
Hide the human and you don't get a higher resolution rate. You get angry customers and a number that lies to you.
To us, handoff is the second tier of a triage system, the backup for when the AI can't resolve a conversation on its own. Its whole job is getting the customer the fastest, best, right answer. Sometimes that's an instant AI reply, and sometimes it's routing straight to a person.
The real benefit of having AI in front is that your humans can react faster to the conversations that do reach them, and give a higher level of care when they get there. Good escalation fires in a few ways: the customer asks for a person, the AI senses frustration, the AI can't answer, or the topic is one you've flagged for a human.
This is also what keeps a resolution number honest (and it's the part most vendors quietly skip). The dangerous version of "autonomous resolution" is when a vendor counts containment or deflection as resolution and quietly makes it hard to reach a human, so the number looks great while customers give up.
We count a conversation as resolved simply when it wasn't escalated to a human, and we're happy for that to be a basic, defensible signal precisely because we make escalation easy. We don't pretend to know a problem was truly solved unless the customer tells us so.
Hiding the human tends to make every reply feel like a vending machine, too, which is its own mistake. Automating the repetitive tail shouldn't make support colder. It hasn't for the customers who do it well, with Edel Optics running 92% AI CSAT, TravelJoy 86%, and YouGarden 78% across 11,785 tickets.
The fix is to make "talk to a person" easy and obvious, wire escalation to fire on frustration and on the topics you don't want automated, and treat handoff as part of how good support is meant to work.
Mistake 6: Measuring deflection instead of resolution
TL;DR: Report resolution, since deflection only means a ticket didn't reach a human while resolution means the problem got solved.
If your weekly AI report leads with deflection rate, I'd worry it's hiding a worse experience than you had before. Deflection only tells you a contact didn't reach a human. It doesn't tell you the customer's problem got solved, and it can climb while CSAT falls, because a customer who gives up in frustration counts as "deflected" too.
Of the three numbers vendors throw around (containment, deflection, resolution), resolution is the one that matters, because it's the only one that implies the issue was actually solved.
Containment names the channel the chat stayed in. Deflection names that a ticket didn't open. Resolution names the outcome the customer actually cares about.
A close cousin of this mistake (and I see it a lot) is chasing response speed. A fast wrong answer is worse than a slightly slower right one, and first-response-time is a vanity metric if nothing gets resolved.
This is why we report resolution and CSAT together. Resolution says the conversation ended without needing a person, and CSAT catches whether it ended well.
A clean before-and-after says it best. TravelJoy went from 24% resolution on Zendesk's own AI to 80% after switching, in their Head of Customer Service's words:
"We're now achieving an impressive 76% AI resolution rate, versus just 24% before." Alan Pugh, Head of Customer Service at TravelJoy.
That's a resolution number, and a deflection one would have told you far less.
The fix is to make resolution rate plus CSAT the headline of your weekly review, and keep deflection as background context.
Mistake 7: Treating it as set-and-forget (expecting it to improve without you)
TL;DR: AI support is a system you operate, so keep climbing the ladder from knowledge to APIs to Tasks. Day one is the worst it'll ever be.
This mistake contains the other six, and I see it more than any of the others. AI support is not a set-and-forget product, however hard it's sold as one. Day one is the worst it'll ever be, and it only gets better if you keep working on it.
The improvement comes in a predictable ladder, and each rung is a different kind of work. The first rung is knowledge, so connect your help center, website and docs. If your knowledge is good this is quick, often minutes to hours, and you can be live the same day.
The next leap, and usually the biggest, is connecting user data via APIs to your backend (orders, accounts, subscriptions) so the AI can answer "where's my order?" with the real answer. That's typically one to three hours of work per API, depending on your dev team's availability. The final rung is Tasks, which are multi-step procedures that combine knowledge, those APIs, and doing something for the customer, like a refund, a cancellation, or an ID check.
We've seen all three pay off. Some teams connect knowledge and get a big jump straight away.
Others jump 50% or more just from connecting APIs, the way Edel Optics did (we watch customers climb this ladder all the time). And some simply go in week after week, fix what the AI couldn't answer, add what was missing, and double their resolution over time.
The mindset that makes the difference is simple. Any effort you put in benefits every future customer the AI serves, so it compounds. Realistically the bulk of the work is in the first month (almost everyone is live and replying directly within a month), and after that it's roughly thirty minutes to an hour a week.
The fix is the whole point of this post. Operate it: look at what it isn't resolving, find the highest-impact gaps, and work through them, week on week.
What this looks like in real rollouts
TL;DR: My AskAI customers who did the operating work run 66% to 82% resolution, from TravelJoy's jump to 80% to Edel Optics hitting 79% after connecting their data.
None of this is theory. The numbers below are from My AskAI customers, and every one of them did the operating work this post argues for.
- TravelJoy went from 24% resolution on Zendesk's own AI to 80% after switching, at 86% AI CSAT, saving around 193 hours a month.
- Edel Optics climbed from 20-30% to about 79% by connecting their User Data API, at 92% AI CSAT across 4,067 tickets.
- Kriptomat rejected Intercom Fin at $0.99 per resolution and reached 62% resolution on a per-ticket model. Freecash did the same on its way to 82%.
- RecruitCRM runs 68% resolution (up from roughly 35% at go-live) at 75% CSAT, leaning on the answer-tracing audit trail to keep quality honest.
- YouGarden handles 11,785 tickets at 66% resolution (peaking around 82%), saving roughly 965 hours a month.
The common thread is simple. Each team kept climbing the ladder instead of switching the AI on and walking away.
What to do this week
TL;DR: Start with one rung this week: clean your knowledge, read 20 answers, switch your headline metric to resolution, then connect an API.
You can start operating your AI agent properly without launching a big project. Work down the ladder, roughly a rung at a time.
- Aim it (this week). Connect and tidy your knowledge, then grade your top 10 ticket types: can the AI answer each from what you've given it? If your docs are good, you can be live in hours. (~1-2 hours.)
- Check how it answers. Open 20 AI replies and ask two things of each: why did it answer this, and where could a customer reach a human? (~30 minutes.)
- Fix your scorecard. Swap the headline of your weekly report from deflection to resolution rate plus CSAT. (~15 minutes.)
- Climb the next rung, connect user data. Wire up an API to your backend for the highest-volume "where's my order / what plan am I on" questions. This is where the biggest leaps come from. (~1-3 hours per API with dev help.)
- Then build a Task. Pick the two or three high-volume, low-resolution areas and build a procedure for each. Then keep going, a rung at a time.
When these aren't mistakes
TL;DR: A few teams are right to track deflection, run a narrow scripted bot, or stay on native helpdesk AI. The point is to choose on purpose.
A few honest exceptions, because no framework fits every team, and we'd rather be straight with you about them. Deflection is a perfectly reasonable headline metric for a pure triage-first team that routes almost everything and measures CSAT on the human side. A scripted bot is fine if you genuinely only have one or two narrow, unchanging flows.
If you're already at a high resolution rate on day one because your docs are excellent, there's less ongoing lift to chase, since the curve has a ceiling. And if your priority is the simplest possible procurement and you trust your incumbent, your helpdesk's own native AI may be the path of least resistance, even if it costs more (we'd honestly rather you picked that than fought your own stack).
There's no single right answer here for everyone. The point of naming the mistakes is to make the choice on purpose.
The takeaway
TL;DR: Almost every AI support failure comes from treating it as set-and-forget. Operate it across three stages and it only gets better from day one.
Day one is the worst your AI support agent will ever be. Almost every common mistake (thin knowledge, the unforecastable bill, the scripted bot, no guardrails, the hidden human, the wrong metric) is a symptom of treating it as set-and-forget, when it's really a system you operate across three stages: what you aim it at, how you let it answer, and how you keep score.
So pick one rung this week. Swap deflection for resolution in your weekly review, or connect one API, or just open twenty answers and read them.
Any effort you put in benefits every customer the AI serves after, so keep going, week on week. That mindset, more than any single feature, is what I've seen separate the rollouts that double their resolution from the ones that stall.
FAQs
How do I know if I'm making these AI customer service mistakes?
Run the three-stage check. Look at what you aim it at (is your knowledge current, is it connected to live data?), how it answers (can a customer reach a human, can you trace why it answered?), and how you keep score (are you reporting resolution, or just deflection?). In our experience, if your weekly report leads with deflection and nobody has touched the knowledge base since launch, you're almost certainly making at least three of them.
What's the single biggest mistake in AI customer service?
Treating it as set-and-forget. Day one is the worst your agent will ever be, and the teams who switch it on and walk away freeze it there. The ones we see double their resolution are the ones who keep connecting data, fixing gaps and adding workflows as they go.
Do I have to replace my helpdesk to use AI customer service?
No, and you shouldn't have to. The better model is an AI agent that layers on top of the helpdesk you already use, keeping your tickets, tags, macros and routing intact. We built My AskAI to install inside Zendesk, Intercom, Freshdesk, Gorgias or HubSpot rather than replace them, so you swap out the native AI while keeping your whole stack.
How do I stop AI giving wrong answers to customers?
Start with knowledge hygiene, since most wrong answers are stale or ambiguous docs rather than the model inventing things. Then set guidance rules, test in internal-notes mode before going live, and keep an audit trail so your team can open any conversation and see exactly what the AI used. Even self-learning needs a human eye on what it adds (we've seen the odd transcription slip prove the point).
What's the difference between an AI chatbot and an AI agent for support?
A chatbot follows a fixed script or decision tree and dead-ends on anything it wasn't built for. An AI agent understands a question it's never seen, finds the answer, and where needed takes the action (looking up an order, processing a refund) by calling your systems. The quick test we'd suggest: can it handle a question that's in no script, and can it take an action or only answer?
Should I track deflection or resolution?
Resolution, every time. It's the only one of the common metrics that implies the customer's problem was actually solved. Deflection just means a ticket didn't reach a human, which can happen because they gave up, so we'd always report resolution rate alongside CSAT to know the conversation both ended and ended well.
How long until my AI agent actually gets good?
Almost everyone is live and replying directly within a month, often the same day if they start from good knowledge. The biggest gains come in the first few weeks as you connect data and fix gaps, then it settles into an ongoing climb of roughly thirty minutes to an hour a week. Remember, day one is the worst it'll ever be, and from what we've seen the curve only goes up from there if you keep operating it.
