Design RAG retrieval and reranking for support questions

The support case

Establish the question before searching

Retrieval quality starts before any search runs. Use the recent conversation to work out the subject, the time period and what is still unresolved. Keep what the customer actually said separate from what you infer from it: “my order” is a statement, while “they mean order ORD-204” is a guess until the customer or a trusted record confirms it.

If a missing detail would change the next action, ask a targeted question instead of guessing. “Are you waiting for the parcel or the refund?” gets you a useful answer in one turn. A broad “Can you tell me more?” makes the customer do the work of figuring out what you need.

When the question is clear enough to search, search for the stated issue and for the reasonable alternatives, without turning one of those alternatives into a fact. Write the search request from the conversation, not just the last message, and keep literal details such as order numbers, product names and error codes exactly as the customer typed them.

Use the search method that fits the question

Support questions come in two broad shapes, and each suits a different kind of search. Keyword search matches the exact words in the query, which is what you want for an error code like “E-1042”, a SKU or a named plan. Semantic search uses embeddings, numeric representations of meaning, so it can match “my card keeps getting declined” to an article titled “Payment failures” even though the words differ.

Most support traffic mixes both, so test both. Hybrid search runs the two methods and merges the results, and it often helps when one message contains an exact identifier and a loose description of the symptom. Measure it on your own questions before adopting it, because it also adds tuning work and another thing to debug.

Apply access filters before anything reaches the model

The retrieval request should carry the audience and account permissions that your application has verified, not ones the model or the customer supplied. Use them to filter out restricted sources before the model sees any text. If you give the model a restricted fact and then tell it not to mention it, you are relying on the model’s obedience, and a paraphrase or summary can still leak it. Filtering first means the fact is simply not there to leak.

It also helps to keep two kinds of private content apart. An approved internal instruction, such as “transfer billing disputes over $500 to the finance queue”, is something the agent is meant to use, even though the customer never sees the instruction itself. A restricted account document may contain information this visitor must not receive at all. Record those as separate classes in your source register so the filter treats them differently.

Treat reranking as a measured choice

A reranker is a second model that takes the candidates your search returned and re-scores each one against the question, so the most useful passages move to the top. That can make a real difference when search finds the right article but ranks it fifth. It cannot fix a source that does not exist, an access-control mistake or a query so poor that none of the candidates are useful.

To decide whether you need one, compare a baseline without reranking against a configuration with it, using the same documents and a set of held-out questions the team did not use for tuning. Look at whether the needed evidence reaches the top results and whether the final answers improve, then weigh that against the extra latency and cost per request.

Options include hosted rerankers from Cohere and Voyage AI by MongoDB, and open models such as BGE through the FlagEmbedding library. These are examples to evaluate, not a statement about which models My AskAI uses. Before you commit, check the specific model’s language coverage, input limits, license, hosting requirements and current terms.

Give the answer step a bounded set of evidence

Pass each piece of evidence with its title, where it came from, when it was last updated and enough surrounding text to keep its conditions attached. An answer can fail because it gets too little context: a chunk that says “refunds take 5 days” without the following sentence, “except for international orders”, will produce a confident wrong reply. Adding every retrieved chunk causes the opposite problem, burying the one rule that matters under loosely related material.

Set a context budget, remove duplicates and measure what gets cut. Treat an empty search, a truncated result, conflicting sources and a failed search as different states, because each calls for a different reply. “I couldn’t find anything on that” is honest after an empty search, but misleading after a search that errored.

When sources conflict, apply your defined source priority or ask the content owner to resolve it. Don’t let the model quietly pick whichever policy is most convenient. Relevance scores are not a verdict either: a low score is not proof the answer is wrong, and a high score is not proof the fact is current.

Test retrieval separately from the final answer

A good final reply can hide weak retrieval, and a bad reply can come from good retrieval used badly. Evaluate the two steps separately. For each test question, label which source should be found, check whether search found it, then check whether the final reply used it correctly.

Build the test set from the cases that break retrieval in practice: misspellings, short follow-ups like “and the other one?”, exact codes, questions in other languages, questions with no answer in your content, policies that recently changed and questions whose only matching source is restricted. Re-run the set whenever you change sources, embeddings or ranking.

Keep it working

Launch is the start of the work, not the end. These are the recurring jobs this part of the agent needs, and who usually owns them.