Mike HeapMike is an experienced Product Manager who focuses on all the “non-development” areas of My AskAI, from finance and customer success to product design, copywriting, testing and more.
Table of Contents
- Who is this guide for?
- What does it mean to be ‘AI-first’ with Customer Service?
- What does it not mean?
- ‘AI-first’ Customer Service won’t work for us…
- What are the benefits of AI Customer Service agents?
- Are they better or worse at anything in particular?
- Where is AI-first Customer Service headed?
- 1. Evaluate Your Current Set-up
- Where does the majority of your support volume come from?
- What types of questions do you get today?
- Where is your business knowledge? (and is it ready?)
- What does your current tool or platform let you do?
- 2. Setting Your AI Objectives
- What are you trying to achieve with your AI customer service agent?
- 3. Choosing an AI Agent Provider
- Where do you find these AI customer service agent providers?
- What knowledge can you add?
- Where can you use it?
- How can you use the AI agent?
- Does it hand over to a person if the AI can’t answer?
- How can you improve my AI agent over time?
- What is the answer quality like?
- Which models does it use?
- How does the pricing work?
- What support will you get?
- How secure is it?
- 4. Testing AI Agent Providers
- Phase 1 - Setting up
- Phase 2 - Asking questions
- Gathering test questions
- Scoring test questions
- Making a decision
Created time
Sep 16, 2024 01:32 PM
Ready for draft
Ready for draft
Author
Ecomm?
Relevant with changes
Image
Publish date
Sep 16, 2024
Generate
Generate
Slug
getting-started-with-ai-first-customer-service
Featured
Featured
Type
Article
Ready to Publish
Ready to Publish
AI customer service agents have well and truly landed.
Forget about all the ‘AI’ chatbots you have used previously.
These AI agents are a new breed, taking advantage of all the latest developments in generative AI, popularised by ChatGPT, to talk to customers just like a human agent would.
But, instead of training your human agent over weeks or months, you can train them pretty much instantly, on all your company documentation.
Forget spending hours creating chat flows and frustrating customers.
This time, it really is different.
In this guide, I’ll show you how to go from an AI novice to someone capable of making a decision for their company about which AI vendor will best fit their business’s customer service needs.
Who is this guide for?
If you’re reading this guide you are probably either:
- A Customer Service Leader (Head or Director of) OR
- Leading a support transformation project in your organization OR
- An enthusiastic AI early adopter, Operations Manager or Product Manager trying to help your Customer Service team out
Maybe you have been told by your CEO that “we need to be using AI!” or maybe you are just an enterprising individual, either way, I’ll take you through the common questions and pitfalls to help you on your way to choosing your first AI agent.
AI agents can work across pretty much any industry, from healthcare to SaaS products, it’s just about finding the right one for you.
What does it mean to be ‘AI-first’ with Customer Service?
Being ‘AI-first’ is about empowering your users to self-serve with AI, getting them better answers to their questions, faster so they can unlock the value of your product.
It means their first contact when they have a question will be your AI agent, trained on all your product’s knowledge, available to them 24/7 and responding in seconds.
What does it not mean?
Being ‘AI-first’ doesn’t mean that you will no longer be speaking to your customers.
Nor does it mean that you are ‘cost-cutting’ or providing a worse experience for your customers.
AI has come on leaps and bounds in the last few years, but it still isn’t at the point where you can ‘set it and forget it’.
You’ll still need people to provide support to your users.
But hopefully, the support they will provide will be for the more complex issues and for the higher-value users.
Plus, as an added bonus, it’ll mean they spend less time responding to the monotonous, tier 1 queries that could likely be solved by taking a look at your knowledge articles.
‘AI-first’ Customer Service won’t work for us…
We’re still in the very early stages with true AI agents, and so, understandably there are a few common objections that tend to arise (either from you or your boss), let’s talk through them:
- “We’ll do it later when AI is more advanced and reliable”
Like with most things in life, the sooner you start doing the faster you learn and the better you can make the experience for your customers.
Improvements in AI won’t fix most of the things that will cause poor AI agent performance, like gaps in your knowledge base or how your users will familiarise themselves with the AI agent.
There are lots of ways you can start passively using AI agents right now to assist your human agents, while you get things up to the standards you would be comfortable with deploying en masse.
Don’t wait, just start, future you will thank you.
- “AI makes things up”/”AI can’t be trusted”
While stories of AI ‘hallucinations’ telling people to eat rocks or lawyers presenting made-up cases in court populate the headlines, the reality, especially for AI agents is very different.
These stories come from general-purpose chatbots that are required to answer just about any question.
Your AI agent will likely be trained on a very specific, finite set of information you provide to it, meaning it will only answer questions using that information, nothing else.
Most AI agents will have some form of guardrails to prevent them from ‘hallucinating’, referring to competitors’ products or answering outside of their knowledge base (if they don’t you should probably look elsewhere…).
Incorrect responses generally result as a consequence of unclear documentation provided to the AI, the AI agent is in effect a ‘mirror’ being held up to your knowledge base, so if its answers are wrong there is a good chance a person reading them would also have misunderstood.
- “Our customers won’t like it”/”Our customers will leave us”
A portion of people hate change, even when it benefits them.
They are also often the ‘loud minority’ of customers rather than the ‘silent majority’.
This shouldn’t prevent you from doing what is in the interest of your entire customer base.
People don’t hate AI agents (chatbots) they hate bad AI chatbots.
They are experiencing the hangover of 10-15 years of poorly executed, non-generative AI agents, so who can blame them?
But not utilizing AI isn’t the answer, like most things you have to introduce it to them carefully and build up trust in it over time.
The longer you wait, the longer it’ll take.
- “We don’t have the time to set it up”
Gone are the days of painstakingly mapping out customer queries and converting them into chatflows.
Now to set up an AI agent it is often as easy adding a URL, connecting a knowledge base and a few minutes later, asking it a few questions.
It really can be that simple to get up and running.
Over time you can improve the set-up, refine the training knowledge, increase the deflection rate, but you’ll get 70-80% of the way there in less than a day.
- “It doesn’t feel very personal”
People use products to solve their problems.
People don’t sign up for products or services to make friends with the people who run them.
This doesn’t mean an AI agent should be impolite, or that the customer shouldn’t feel listened to and be empathised with.
But it does mean people will be happy with any agent who can answer their questions, fast, and allow them to get on with their day or job.
AI agents are now much better at sounding empathetic, and reacting to customers’ frustrations.
And the time saved will allow you to focus on other ways to add that ‘personal touch’ to their user experience.
- “It’ll be expensive”
- Pay per successful resolution or deflection
- Pay per question, pay per conversation or
- Pay per agent
There are AI agent pricing models suited to most businesses nowadays:
to name just a few.
In almost every case, it will be cheaper than using a person to do the same task, with prices ranging from $0.05 to $1.99 per interaction depending on the AI agent you choose.
The upfront ‘cost’ of setup is also negligible (see 4).
Add to this that costs will likely continue to fall, and AI competence will continue to improve, making the return on investment today the worst you will likely ever experience.
What are the benefits of AI Customer Service agents?
An AI agent can have multiple benefits to your business, even if you are only just starting out.
Not convinced? How about these for starters:
- Handle more queries, faster
One of the biggest benefits of AI customer service agents is their near-instant response times. They can theoretically respond to all your customers, simultaneously, in seconds.
Just think how many human agents you would need to do this, or even ensure a sub-minute response time to all queries.
It just wouldn’t be economical.
AI agents scale up and down with demand, meaning they are there only as and when you need them, whether you suddenly get a spike in traffic or it’s a quiet holiday.
Even if your AI agent is unable to answer every question, each question it does answer is one less your human agents have to.
- 24/7 availability
Humans need sleep (whether you like it or not).
But customers for most software businesses can come from all over the world.
Why limit yourself to a single location because of slow support replies when you can use an AI agent that doesn’t sleep?
- Reduced costs
- High-quality, skilled agents
- Who are also very cheap
- Dealing with relatively simple tickets
- Are also available 21 working days/mo
- Don’t need time off
- Don’t need training days
- Don’t need to attend meetings
- …
However cheap people are, technology is and will almost always be cheaper.
Instead of driving down the costs of your people, if you assume a skilled human agent can deal with 50-100 tickets per day, and, at their cheapest is c. $1,500/mo, then (assuming 21 working days/mo), their cost per ticket will be somewhere between $0.71 and $1.43.
But for this you need:
Do you see where I am going with this?
Your best-case, cheapest situation is somewhere around the $1 mark per ticket when using people.
This is the price of the most expensive AI.
AI agents on the market today are pretty much all somewhere between 2-10x cheaper per ticket than this.
- Consistent and accurate responses
People are great, we love people.
But people are not machines.
We are not built for consistency and memory.
We say things in different ways depending on how we feel and we forget things from time to time.
Instead of training people to be more “machine”-like with their responses by using shortcuts, quick replies and canned responses, why not cut the middle-person and just… use a machine (or an AI agent, in this case) to answer questions consistently and accurately?
- Improved customer satisfaction and experience
You may raise your eyebrow at this one.
Rightfully so, in some circumstances, people don’t like dealing with ‘bots’ (largely because of bad previous experiences).
But if they are set up well and improved upon over time, they do improve customer experience.
Who doesn’t like getting fast, helpful answers to questions so you can get on with your day?
There may be a small (but loud) minority of users initially who speak out against your new “robotic” approach, but they are also probably the same users who complain when it takes you hours to respond to their basic questions that you have answered in your docs.
Focus on the largely silent majority, focus on the data and commit to improving each aspect of your AI agent’s performance.
- Freeing up human agents for more complex issues
It’s going to be a long time before AI is capable of taking entire jobs away from people (if it ever will).
AI is better at assuming ‘tasks’ from people.
Let it take on the ‘simple’, repetitive tasks, queries and questions from users.
Leave people to deal with the more complex, bespoke, edge-case issues that require more consideration, empathy and things only a person can provide.
Your team will thank you for making their role far less monotonous.
Are they better or worse at anything in particular?
There are some things AI agents are very good for, others it is ok at and a few things you probably should still be doing yourself:
Good
- You have detailed documentation that your AI agent can be trained on
- Your queries can often be answered using your documentation without needing to access a user’s account or other databases
Not so good
- You have a ‘long tail’ of unique, account-specific or trouble-shooting type questions that take a person multiple steps to resolve
- Your users expect a high-touch, personal service as part of your offering e.g. a concierge service
- You have very specific, defined and deterministic actions you need a user to perform each time they use the live chat or you need to collect specific data
There are of course ways (and products) that each of these ‘Not so good’ examples can be resolved, they are just not usually offered ‘out of the box’ with most AI agent products.
Also, it is worth bearing in mind that any question, instantly answered, that doesn’t require a human, is good for both the customer and for your businesses as it saves them time and you money.
Even if your AI resolution or deflection rate is low double digits, it can still have a positive impact on your business.
Where is AI-first Customer Service headed?
In the grand scheme of AI customer service, we are probably only on Day 2 or 3 in terms of progress.
We are stepping away from drag-and-drop chat flow builders and have moved on to using cutting-edge generative AI.
But there is still a long way to go.
It won’t be long before we will be able to use AI agents to:
- Easily and seamlessly integrate any data source to give our AI agents complete knowledge of our business and users to answer their questions.
- Have our AI agents work alongside users to identify, debug and even fix their issues live on their screens, maybe even before they ask.
- Allow our AI agents to autonomously perform actions within our platforms to solve issues and perform account actions (refunds, upgrades and more).
- Instantaneously generate live documentation, perpetually, as you update your product screens.
Excited?
Now, let’s get you started on your journey to AI Customer Service.
1. Evaluate Your Current Set-up
The first question you should be asking yourself before moving to AI-first customer service is:
“What does my current set-up look like?”
Before finding out where you are going, we first need to understand where you are.
Let’s break this question down into a few further questions to make it easier:
Where does the majority of your support volume come from?
You already have existing support tickets but where do the majority come to you from? Is it:
- Calls
- A combination of the above?
Different AI tools specialise or offer integrations with different channels and platforms.
While some vendors offer the option to build custom integrations as part of Enterprise plans, most won’t, so be sure that the vendors you choose work with your current support setup. Unless…
You may also want to take this as good an opportunity as any to switch support channels.
Maybe you have been using email primarily for the last decade but now think it is the time to finally shift to live chat tools.
It might sound drastic, but start by considering what would be the best set-up for your customers and work backwards.
It might have been when you decided on your current channels that you did so partly so you were able to cope with the increasing volume of tickets you were receiving as you scaled.
But with AI agents it may mean you can scale your support further, providing more instant responses with live chat or AI calls.
What types of questions do you get today?
You know your inbox as well as anyone but it still never harms to dive in and take a sample to check you still have your finger on the pulse.
Take a typical, day, week or month and randomly pick a few hundred tickets and copy the conversations into a spreadsheet.
For each ticket ask yourself “Could this have been answered…”:
- Soley using your currently available documentation
- Using documentation you could practically write (but haven’t)
- With access to other business knowledge (account details etc)
Tickets in buckets a. and b. are very good candidates for AI agents and the proportion of tickets in a. overall will give you a good idea of the % of tickets you might be aiming for as being ‘resolved’ by an AI agent without human input required.
Where is your business knowledge? (and is it ready?)
Your AI agent’s performance will depend considerably on the information (and the quality of that information) you provide to it for ‘training’.
The more it knows about your business the better it will likely be able to answer your customer’s questions.
As a general rule of thumb, the easier it is for a person to understand and read your documentation, the better an AI will be able to understand it.
Different AI agent tools have differing abilities to ingest information from different sources so this should definitely be a consideration upfront and also for future training (you may not use some sources right now, but this could change).
Common knowledge sources can include:
- Websites (check for how often these are updated and whether new pages are added)
- Knowledge tools (Google Drive, Notion, Dropbox, Coda, Confluence, SharePoint etc)
- Ticketing platforms* (Intercom, Zendesk, Freshdesk)
- Communication tools* (Gmail, Slack, Outlook)
With each of these, you may also want to consider:
- Which file types is my knowledge stored in?
- How often are they updated and are new items synced automatically?
This is also a very good time to take a good hard look at your documentation and consider whether it is ‘ready’ for an AI agent.
Luckily, most tools provide ways of identifying ways to improve your documentation over time as questions come in but they will at least need some up-to-date documentation to start with.
*A word of caution
While on paper it sounds like a fantastic idea to upload all of your previous tickets or internal conversations as ‘training’ data to your AI agent, in practice this is often not quite so good an idea. Not only does the large volume of tickets often make it difficult to manage, the quality or responses across agents can vary considerably and it is incredibly difficult to manage dated information. I would generally advocate for using a ‘source of truth’ for your AI agent’s knowledge - one that is easily accessible and updateable by members of your team.
What does your current tool or platform let you do?
Before going looking around for your fancy new AI agent, it is also always worth finding out what your current provider can offer as a benchmark for pricing and features.
Some of the bigger companies in the space have some excellent AI offerings already, although bear in mind the larger companies with the best offerings are usually priced as such and you can often get the vast majority of benefits and features by looking elsewhere.
2. Setting Your AI Objectives
While it may be ‘sexy’ to jump on the AI bandwagon or be seduced by a Zendesk salesperson, before getting into bed with a tool you may later regret it is sensible to outline what you are trying to achieve with your new AI customer service agent.
Different tools have different strengths and AI agents will impact different businesses in different ways so make sure you are clear on what you want to get out of your AI agent and, ideally, the cost associated with that improvement.
What are you trying to achieve with your AI customer service agent?
Here are a few ideas for AI agent objectives you might want to consider:
- Reducing costs
- Reducing time spent on support
- Spending more time elsewhere (not on reactive support)
- Improving response times
- Enhancing customer experience
- Increasing resolution rate
- Gathering customer insights
- Ensuring scalability
All very reasonable (and achievable) objectives when deploying an AI agent.
The next step is then to try and turn these objectives into quantitative, specific goals, using readily measurable metrics, such as improving new AI metrics like:
- AI Resolution or Deflection rate (what % of support interactions are resolved solely by the AI agent)
- Bot Engagement rate (what % of support interactions is the bot used for)
Or more well-known customer support metrics like:
- Average Handling Time (AHT)
- First Contact Resolution (FCR) rate
- Customer Satisfaction (CSAT) score
- Net Promoter Score (NPS)
- Ticket Volume
- Response Time
- Resolution Time TTC
- Customer Effort Score (CES)
- Cost per Contact
- Escalation Rate
- Abandonment Rate
- Agent Utilization
Start by getting your benchmark metrics today over a reasonable period for your business. These can then be used as your baseline metrics for when you introduce your AI agent.
They will also be handy for presenting to vendors, you can ask them if they have examples of case studies from their clients and how they impacted the metrics that matter most to you and your business.
3. Choosing an AI Agent Provider
Now it’s time to choose your shortlist of AI customer service agent providers to start testing with.
I’d always recommend testing with a handful of vendors as they will all have a slightly different setup in terms of how they answer questions, some will work to your benefit, and others will work against your setup.
Just because one provider is optimal for one company, does not mean it is optimal for yours (ever hear the phrase “jack of all trades, master of none”?
This is most definitely the case in the world of AI agents, you are often much better suited looking for a specialist AI customer service agent.
Where do you find these AI customer service agent providers?
If you are already using a customer service platform like Zendesk or Intercom, I would start by checking their respective marketplaces searching for something like: “AI support” or “AI chatbot”.
Googling phrases like “AI support agent”, “AI customer support”, “AI customer service” should bring a good list to begin with.
Most products will also create “Alternative to” pages which show comparisons of who they think their most similar competitors are.
While these lists are likely biased they can serve as a good research tool to begin your search.
Your goal here is to find say 5-15 vendors that you can then bring down to a shortlist of 2-5.
Now, the next question is, how do you reduce this list of 5-15 vendors, down to your shortlist?
Here are a few questions to ask yourself (and your vendors) that will help:
What knowledge can you add?
Consider:
- Where your knowledge is currently
- Where it might be in the future (if you are going through any imminent knowledge changes)
- How often knowledge gets polled/updated by the AI agent provider
- How much knowledge you can add (what are the limits?)
- Which file types are compatible
Where can you use it?
This is an easy one - does it connect with the support tool you want to use going forward?
But also consider whether you might also want to provide AI agent responses in other tools:
- Directly from your email inbox
- In your browser (via a Chrome extension or similar)
- On your desktop
- Within native apps
How can you use the AI agent?
There are a few different AI customer service patterns that have emerged over the last few years, so make sure you understand which one(s) your AI agent can do for you (and don’t forget that you may only want a ‘co-pilot’ today, but this may change in the next few years…)
- Direct replies within a messenger/live chat/ticketing platforms - One of the predominant AI patterns is to allow the AI agent to reply like (but not as, usually) a human agent would, directly in the live chat widget or ticketing platform, usually providing links to relevant resources when required
- Indirect, note or comment replies within an inbox or messenger - Most companies aren’t prepared to put their AI agent live instantly, so they often opt to use it internally initially, to understand how it might respond and so they can work on improving responses before setting live to directly respond. These replies often occur in-line, in response to user queries, but are only visible to the agent. They can are then used either by copying and pasting responses or as inspiration for the agent’s response.
- ‘Co-pilots’ - One of the more popular initial ways of using AI agents is to use it as a co-pilot of sorts for existing human agents so they can provide better answers, faster. These will often be built into an agent’s inbox or dashboard and will either: suggest a response automatically for the agent, let the agent type a question to get a response or will allow the agent to generate a response by clicking a button on a ticket or conversation.
This is one of the biggest differences between different AI agent providers so make sure you find a provider whose pattern mimics the way you and your team work.
Does it hand over to a person if the AI can’t answer?
This should be table stakes for any AI agent.
No AI agent on the market today is in a place to 100% autonomously provide customer service with no input from a person.
This means that any AI agent tool you look at must provide a seamless way for tickets or conversations the AI can’t answer to be ‘handed over’ to a person to ensure that your customers will always get an answer.
It is also vital that the user experience is as easy as possible for a customer to speak to a human agent when they need to.
It is therefore worth asking your vendors:
When the AI is unable to assist, what are the ways and how does the customer speak to a person?
How can you improve my AI agent over time?
The day you first try out your AI agent will be the worst-performing it should ever be.
There are 2 major components to AI customer service agent performance:
- Algorithm performance - how well the AI service provider has set up your AI agent, this isn’t really in your control
- Knowledge used to train - this is the information you have provided to the AI agent to answer questions, this is very much in your control.
To improve #2 you will need an AI agent tool which can ideally do some or all of the following:
- Show you the common questions it is being asked
- Show you the questions that it can’t answer
- Show you the questions that are triggering handovers
- Show you the most used articles or pieces of knowledge
- Give you an easy way to generate relevant new knowledge
- Let you add ‘custom answers’ or ‘knowledge snippets’ to fill any gaps
- Allow you to add answers from correctly answered tickets by your human agents
If you don’t have at least some of these options available to you, it will make it very difficult for you to improve that all-important AI deflection or resolution rate over time, however good your AI agent provider does with #1.
What is the answer quality like?
The only way to accurately test this is to test.
I’ll go through this in more detail in Step 4 but in short, you will need a list of real questions, tickets or conversations that have been recently asked to your support.
I’d recommend somewhere between 50 and 250 to test with, depending on the variety of questions and complexity of your documentation (the more and the more complex, the closer to 250 you’ll want).
Any good AI agent provider should allow you to test this set of questions free of charge to see what responses you could expect to get from your new AI agent.
See Step 4 for more on testing answer quality.
Which models does it use?
AI models change all the time, you should however check that which AI agent provider you speak to is at least using the highest quality model from their respective model provider.
For instance, for OpenAI, the best-performing model right now is their GPT-4o model, don’t settle for anything less!
They may also tell you it is using a ‘fine-tuned’ model, which can vary considerably in interpretation.
It should also be noted that the best models can give the worst answers when set up incorrectly and the worst models can give market-leading answers and responses if they are set up by a team who knows what they are doing.
In short, I would recommend you largely ignore such statements and judge your AI agent solely on its answer quality.
How does the pricing work?
This is where things can get complicated…
Each AI agent provider will likely offer very different, unique feature sets, which can make it quite hard to compare.
However almost all will have pricing that scales according to some kind of usage metric, here are the common ones:
- Price per ‘resolution’ - resolution-based pricing is relatively new and on the face of things sounds great, in theory - you only pay if the AI has helped your customers, and a human hasn’t had to get involved. However, the downsides of this pricing are that:
- Most providers offering this pricing charge $1-2 per resolution, which while objectively not expensive, can quickly add up. You also have little control over your costs month-to-month
- A ‘resolution’ may not actually ‘resolve’ - usually it just means they didn’t ask to talk to a human within a 24-hour period, even if they just dropped off and stopped responding.
- Price per question/conversation/ticket - may be one of the easiest to understand, this pricing is based on how many times your AI agent is called into action. The downside is you get charged whether it helps or not and you pay more for longer conversations. On the plus side, the better your AI agent gets, the cheaper it will cost per solved ticket.
- Credits - a variation on price per question you are allotted a certain number of credits each month which are used depending on things like - which model you use, how many questions are in a conversation etc. These are probably the most opaque form of pricing but that means you may be able to get a deal if you choose wisely and keep track of your usage!
The other ‘usage-based’ elements of pricing that are relatively commonplace are:
- Number of agents/bots - the number of different AI agents you need (you generally only need more than 1 if you are to serve different, isolated knowledge bases to each or you need each agent to behave very differently from one another).
- Content limits - some AI agent providers will stipulate how many web pages or documents you can train your AI on, this can also sometimes be phrased as the number of ‘characters’ in those pieces of content. It is less normal now to have such limits imposed.
- Number of seats - some AI agent providers will charge by the number of users you want to access the AI agent platform.
Aside from the usage-based comparison, you will have to look for features that sound useful to you as part of your comparison.
Common AI agent features you might see included in pricing lists include:
- Ticket creation
- Help centers
- Chatflow builders
But just ensure you are always checking back to your goals and objectives here and aren’t blinded by shiny object syndrome!
What support will you get?
Depending on your plan and the size of the company you are working with the level of support you get can vary massively.
Check the channels they use for support: email, live chat, their own AI agent, calls, video calls, and customer success teams.
And their availability: 24/7, 24/5, 9-5/7, 9-5/6, 9-5/5 (also check time zones).
You want to find the AI agent provider that best fits your needs.
Also don’t underestimate the value in having the provider help you get set up in the first place and check with them to ensure this is part of the service they provide to ensure you aren’t wasting time optimizing your set-up.
How secure is it?
Security should always be a consideration.
While it is highly unlikely that any AI agent provider will use your conversations and tickets for training a model of theirs (or anyone else’s) it is always worth confirming this.
Depending on the size of your business and the customers you work with you may also need to check compliance with regulatory or security standards such as ISO 27001, GDPR or SOC II.
A lot of the time this should be a pretty simple and quick question for them to answer.
4. Testing AI Agent Providers
Hopefully, by now, you have managed to whittle down to a shortlist of 2-5 AI agent providers that you can put to the test against one another.
They should only be vendors you can see yourself working with and who have cleared your minimum requirements bar from step 3.
Now comes the fun part - testing.
Testing comes in 2 phases:
- Setting up
- Asking questions
Phase 1 - Setting up
The first thing you will need to do is add the training data, or knowledge to the respective AI agent tools.
This shouldn’t take more than a few minutes or hours on each, depending on how much content you are to add and where you are adding it from.
A good vendor may even assist or advise you on the best way to add or upload your knowledge if you are unsure.
The way in which knowledge can be uploaded can also impact results - if for example, you are able to use knowledge connectors (like a direct integration with a Zendesk help center) then always opt for these over a website crawl as you are likely to get more complete and accurate results.
You need to make sure that the knowledge you add will be sufficient to answer the questions you are to ask in Phase 2.
If the AI doesn’t have the knowledge it won’t be able to answer your questions.
You should also try to ensure that each of your AI agent tools has the same knowledge available to it (to ensure a fair test).
It will likely not be exactly the same knowledge across your providers if you are using website knowledge as website crawlers all work in different ways, however, each should retrieve materially the same pages, enough for testing.
Phase 2 - Asking questions
Once you have finished setting up your AI agent by providing it with the requisite knowledge, it is time to start figuring out what questions you will ask.
Ultimately, this is the true test of any AI agent, even with the best integrations and feature set in the world, if it can’t answer your customers’ questions well, it won’t be of much use.
Gathering test questions
This is arguably one of the most important steps in your AI-first journey, so here’s what you need to do to gather your test question set:
- Look at real tickets, questions and conversations your support has received recently.
- Do not clean them up, or edit them, just copy and paste them into a spreadsheet.
- Aim for between 50 and 250 to test with, depending on the variety of questions and complexity of your documentation (the more and the more complex, the closer to 250 you’ll want).
- Choose a variety of questions, where possible you want:
- Simple questions
- Complex questions
- Greetings
- Vague questions
- Technical questions
- Questions that only have one answer hidden in your docs
- Questions you wouldn’t want it to answer
- Questions about your competitor
- Questions about things that are off-topic
- Multi-part questions
This should give you a comprehensive question set that you can use across all your vendors.
Once you have this test question set you can ask the questions manually to the AI agent, or any good AI agent provider should allow you to test your set of questions free of charge (and may do it for you) to see what responses you could expect to get from your new AI agent.
Scoring test questions
Once you have your responses back you will need to score them.
Here is what you will want to score your responses on:
- Answer accuracy - did it answer the question correctly?
- Incorrect answers - did it answer any questions incorrectly?
If there are more than a couple of incorrect answers you should probably remove the provider from your shortlist
- Conciseness - AI tends to ramble, but customers don’t tend to like reading long responses so make sure the ones you get are of an appropriate length
- Source accuracy - did it provide what you would deem to be the most relevant sources (links or documents) back in the answer for further reading?
- Answer tone - Does it fit with your company and brand?
- Clarifications - Did it ask follow-up questions if the question was vague?
- Focus - Did it answer anything it shouldn’t, did it go off-topic or talk about competitors?
- Conversation - How did it handle small talk or informal conversation?
Making a decision
Once you have completed your testing (and finished reading this guide 😉), you are probably as educated as you can be about your AI agent decision.
Choose the agent that scores the best in Phase 2 combined with the team or company you feel most confident working with - AI agents aren’t just for Christmas after all and you’ll want to work closely with your vendor to improve your agent’s performance over time.
I hope this guide has been useful for you and I hope you’ll consider My AskAI as part of your selection process like Customer.io, Zeffy, Freecash and Zinc did.
👋🏻
