How do customer service AI chatbots actually work (in 2024)

Created time

Mar 1, 2024 11:27 AM

Author

1. Indexing all your company’s knowledge

Step 1 is always scraping, extracting, or accessing all your company’s knowledge. This could be on your website, Notion workspace or within Google Drive. This content is basically copied, separated into individual paragraphs (called ‘chunks’ in the industry), and converted into a format that can be retrieved very quickly using natural language search. Which basically means, when a question is asked like “How do I connect to the API”, the paragraphs that are brought back are very similar in meaning (intent) to the question. The most relevant paragraph is not the paragraph with the most keyword matches (traditional search). Once all a company’s knowledge has been indexed, we can now look at how a question can be answered.

2. Question from a customer “How do I add a new team member”

This could be a typical question seen in customer support for a SaaS company. Below I’ll describe how an AI model is used to answer this.

3. Retrieve the most relevant information for the question

The user’s question is converted into that same format as the company’s knowledge base and used to do a natural language search. The result of this search will be ~5 relevant paragraphs, from across all of your company’s content, to the user’s question. In this example, the paragraphs will likely include instructions on adding and removing team members

4. Ask an AI model to answer the question from the relevant information

Now that we have ~5 paragraphs that likely contain the answer somewhere within it (along with some irrelevant info, like removing team members) we can pass this to an AI model. Specially we’re talking about a large language model, like ChatGPT. To visualise this, imagine you’re using ChatGPT on the web, and you type in a request like:

You’re an intelligent customer support assistant, please answer the customers questions with the help documentation provided. If you can’t find an answer, say so.

<5 paragraphs of relevant info pasted>

Customer’s question: How do I add a new team member?

This is just a simplified example of the request an AI chatbot will make, but the principles are the same. It’s providing the AI model with some context so that it can answer a really specific question. And importantly, we’re also telling the AI model to not make an answer up if it can’t find a solution in the paragraphs provided. This helps avoid “hallucinations”, also known as lying, by the AI model. Because if you don’t have instructions on how to add team members, you don’t want it making up those instructions.

That’s it.

From this, you can start to understand the things that will affect the output quality of an AI chatbot. This can then help you decide on which company you choose to create your chatbot with

Help documentation / knowledge available. In an answer is not documented somewhere, an AI model will never provide a good answer

AI model. Whether at chatbot is using the latest AI models available, will have a big impact on answer quality.

Chatbot ‘prompt’ or instructions given to the AI model. Small changes in these instructions can have big consequences on answer quality.

User’s question. AI models are very literal and precise, and for that reason, small changes in how a question is phrased can result in quite different answers. This can be helpful and unhelpful.