Mike HeapMike is an experienced Product Manager who focuses on all the “non-development” areas of My AskAI, from finance and customer success to product design, copywriting, testing and more.
Table of Contents
Created time
Sep 16, 2024 01:56 PM
Author
Publish date
Sep 16, 2024
Slug
ai-first-customer-support-testing-ai-agent-providers
Featured
Featured
Type
Article
Ready to Publish
Ready to Publish
Hopefully, by now, you have managed to whittle down to a shortlist of 2-5 AI agent providers that you can put to the test against one another.
They should only be vendors you can see yourself working with and who have cleared your minimum requirements bar fromstep 3.
Now comes the fun part - testing.
Testing comes in 2 phases:
- Setting up
- Asking questions
Phase 1 - Setting up
The first thing you will need to do isadd the training data, or knowledgeto the respective AI agent tools.
This shouldn’t take more than a few minutes or hours on each, depending on how much content you are to add and where you are adding it from.
A good vendormay even assist or advise you on the best way to add or upload your knowledge if you are unsure.
Thewayin which knowledge can be uploaded can also impact results - if for example, you are able to use knowledge connectors (like a direct integration with aZendesk help center) then always opt for these over a website crawl as you are likely to get more complete and accurate results.
You need to make sure that the knowledge you add will be sufficient to answer the questions you are to ask in Phase 2.
If the AI doesn’t have the knowledge it won’t be able to answer your questions.
You should also try to ensure that each of your AI agent tools has the same knowledge available to it (to ensure a fair test).
It will likely not beexactlythe same knowledge across your providers if you are using website knowledge as website crawlers all work in different ways, however, each should retrieve materially the same pages, enough for testing.
Phase 2 - Asking questions
Once you have finished setting up your AI agent by providing it with the requisite knowledge, it is time to start figuring out what questions you will ask.
Ultimately this is the true test of any AI agent, even with the best integrations and feature set in the world, if it can’t answer your customers’ questions well, it won’t be of much use.
Gathering test questions
This is arguably one of the most important steps in your AI-first journey, so here’s what you need to do to gather your test question set:
- Look at real tickets, questions and conversations your support has received recently.
- Do not clean them up, or edit them, just copy and paste them into a spreadsheet.
- Aim for between 50 and 250 to test with, depending on the variety of questions and complexity of your documentation (the more and the more complex, the closer to 250 you’ll want).
- Choose a variety of questions, where possible you want:
- Simple questions
- Complex questions
- Greetings
- Vague questions
- Technical questions
- Questions that only have one answer hidden in your docs
- Questions you wouldn’t want it to answer
- Questions about your competitor
- Questions about things that are off-topic
- Multi-part questions
This should give you a comprehensive question set that you can use across all your vendors.
Once you have this test question set you can ask the questions manually to the AI agent, or anygood AI agent provider should allow you to test your set of questions free of charge(and may do it for you) to see what responses you could expect to get from your new AI agent.
Scoring test questions
Once you have your responses back you will need to score them.
Here is what you will want to score your responses on:
- Answer accuracy- did it answer the question correctly?
- Incorrect answers- did it answer any questions incorrectly?
If there are more than a couple of incorrect answers you should probably remove the provider from your shortlist
- Conciseness- AI tends to ramble, but customers don’t tend to like reading long responses so make sure the ones you get are of an appropriate length
- Source accuracy- did it provide what you would deem to be the most relevant sources (links or documents) back in the answer for further reading?
- Answer tone- Does it fit with your company and brand?
- Clarifications- Did it ask follow-up questions if the question was vague?
- Focus- Did it answer anything it shouldn’t, did it go off-topic or talk about competitors?
- Conversation- How did it handle small talk or informal conversation?
Making a decision
Once you have completed your testing (and finished reading this guide 😉), you are probably as educated as you can be about your AI agent decision.
Choose the agent that scores the best in Phase 2 combined with the team or company you feel most confident working with - AI agents aren’t just for Christmas after all and you’ll want to work closely with your vendor to improve your agent’s performance over time.
I hope this guide has been useful for you and I hope you’ll considerMy AskAIas part of your selection process likeCustomer.io,Zeffy,FreecashandZincdid.
👋🏻
