Evaluate AI support agents with tests and live experiments
The short answer
Use three layers of evaluation: a fast pack of checks for critical behavior on every change, a representative offline comparison for bigger changes, and a controlled production experiment when you have the traffic to justify one. Check the agent’s task and tool decisions as well as its final answers. Record the configuration that actually ran, and calibrate any AI judge against human review.
Use the right layer for the change
No single test answers every question. A quick pack of known failures tells you whether a change broke something important. A larger offline evaluation, run against saved conversations rather than live customers, tells you whether the new version is better overall. A production experiment tells you how real customers respond. Each layer costs more time than the one before it, so match the layer to the size of the change.
| Layer | Purpose | Example checks | Release rule |
|---|---|---|---|
| Fast functional pack | Catch known failures on each change | Permission denial, schema, task selection, handover and stop rules | All agreed critical cases pass |
| Offline evaluation | Compare behavior across representative cases | Accuracy, retrieval, procedure success, language, latency and cost | Predeclared quality and safety gates with uncertainty |
| Production experiment | Measure real outcomes under controlled exposure | Mature resolution, reopens, handover, customer ratings and cost | Stable assignment, active guardrails and a clear stop decision |
Build a small pack that can block a bad release
The functional pack is a regression suite: a set of tests that catch old problems coming back. Start it from real incidents and your most important boundaries. Include an anonymous visitor asking for private account data, a customer correcting a detail partway through a task, a human taking over the conversation, a question with no matching source, an invalid tool argument and a write whose result is unknown.
Use deterministic checks, plain code assertions with a fixed right answer, wherever the application can verify the fact itself. Whether a denied tool call reached the network is a yes-or-no question. You don’t need a model to grade it.
Add labeled cases for task and guidance selection: a near-miss that should not start the task, a question where no task applies and a message that overlaps two tasks. When a rule depends on several turns, test the whole interaction rather than a single message. Model output varies from run to run, so repeat unstable cases several times instead of treating one lucky pass as a guarantee.
The downloadable example cases are a starting set of fixtures. Their expected outcomes are synthetic, and they are not a benchmark or a claim that your agent passes them. Connect them to your own application checks, and add a new case every time you confirm a bug.
Construct an offline dataset you can trust
An offline evaluation is only as good as the conversations in it. Sample by the units that matter to you, such as conversation, customer, topic, channel and language, so that a handful of very active customers don’t dominate the results. Keep difficult and no-answer cases in the set. A dataset made only of questions that were resolved will hide every escalation failure.
For each case, store the context needed to answer, the expected policy or action, the tool fixtures the agent may use and a short reason for the label. Treat historic human replies as evidence to review, not automatic ground truth, because past agents made mistakes too. Remove private data, and keep a fixed held-out set that is never used to tune the prompt, so your score measures the agent and not how well it memorized the examples.
For each comparison, freeze the dataset, model, prompt, tool schema and knowledge versions so the only difference is the one you are testing. Grow to a few thousand examples when your traffic and question variety support it, but size alone doesn’t make a test representative. Report which segments are covered, what was excluded and how uncertain the result is, allowing for the fact that repeated conversations from the same customer are not independent.
Check the judge and the model that actually answered
Many teams use an LLM judge, a model that grades the agent’s answers against written criteria, because it scales far beyond what people can review. Score factual accuracy, task success, permission safety, handover and communication separately, so a fluent answer can’t make up for an unauthorized action. Use the same judge configuration for every version you compare, and check its labels against a sample that people have reviewed.
Judges have their own failure modes. Score the same answers twice, swap the order in which two answers are shown and look closely at cases where the judge and the human reviewers disagree. Decide in advance how much drift you will accept, and keep unscored cases and failure reasons visible in the report. If you change the judge, score a bridge sample with both the old and new versions, so a jump in the numbers isn’t mistaken for an improvement in the agent.
Record which model actually answered each request, because providers and gateways can fall back to a different model when one is busy. In a pure model comparison, prevent silent substitution or mark substituted runs as invalid. In a comparison of the whole service, fallbacks are part of what you are testing, but report how often they happened. Before you send paid requests, set an attempt limit and a spending ceiling for the run.
Run a production A/B test with one responding arm
In an A/B test, each conversation is assigned to one version (an arm) and you compare outcomes between them. Assign the version by conversation, or by customer when the same person is likely to come back and see both. Only one arm may send replies or execute writes for any conversation. A shadow candidate, one that runs silently alongside the live version, can generate answers for analysis using safe fixtures, but it must never duplicate a customer action.
Before launch, choose how long you will wait for each outcome, how many conversations you need, which guardrail metrics will stop the test and who makes that call. During the test, check that the arms are balanced and that each ran the configuration you intended. Analyze results by the arm each conversation was assigned to, even when a fallback or failure occurred, and record those events. Wait for outcomes to mature before you compare resolution or reopens, since a conversation that looks resolved today may reopen next week.
Don’t run an A/B test just because you can. With low traffic, a test may never collect enough conversations to tell the versions apart. In that case, rely on offline evidence and a controlled rollout, and run an experiment once you can collect enough valid production data to answer the question you care about.
Ask AI to explain this guide
Build decisions and checks
Use these decisions to turn the worked case into your own implementation. A check is complete only when you have evidence for the stated result.
Run a fast functional test pack
Required. A small wording change can break a tool call or stop a handover from happening.
Build. Keep a small suite of known failures and critical behaviors. Check output schema, tool arguments, permissions, state changes, language, stop rules and delivery. Use deterministic checks where you can, and repeat runs where model behavior varies.
Check. Run the suite on every prompt, tool or configuration change, and pass every release-blocking case. Cover which task and guidance the agent selects in critical cases, not just final-answer quality.
Build a representative offline evaluation
Required. A handful of easy examples gives an optimiztic score, and historical answers may be wrong too.
Build. Start with a curated set, then grow to a few thousand production-like cases when volume allows. Include conversation context, languages, hard cases, no-answer cases and tool outcomes. Keep tuning data apart from a held-out set, and redact private data. Decide what counts as one sample, and don't let repeat users dominate. Cap attempts and spend. Record the model and route that actually produced each result. For a pure model comparison, turn off model substitution or mark substituted runs invalid. For a full-service comparison, keep fallbacks on and report them.
Check. Report coverage, failures and results by segment. Freeze the dataset, prompt, model, tool schema and knowledge versions. Set acceptance gates before the run. Report exclusions, how concentrated results are among users, variation between repeat runs, and uncertainty suited to the sampling unit.
Test the judge as well as the agent
Required. A model judge (a model that scores answers) can prefer style over correctness or score the same answer differently between runs.
Build. Define separate criteria for factual accuracy, task success, permission safety, handover and communication. Compare judge scores with human labels. Check repeatability and whether answer order sways the score. Use one shared judge configuration across all experiment arms.
Check. Before testing, set acceptable levels of disagreement with humans, repeat variation and order bias. After a redaction check, share disagreement cases internally and show what went unscored. When you change the judge, score a bridge sample with both versions.
Run a controlled production A/B test
When it applies. A live experiment can mix versions within one conversation or favor one traffic segment.
Build. Assign each conversation to a fixed arm (or each customer, when repeat exposure matters), with only one variant responding. Log the assignment and the model and configuration actually used. Set sample size, quality guardrails and stop rules before launch. Record fallbacks and analyze by original assignment.
Check. Check that arms are balanced, traffic never crosses between them, outcomes have matured and uncertainty is reported. A shadow candidate can't make writes or send replies.
Test the whole service under failure
Required. A high answer score says little about duplicate refunds, lost webhooks or failed human assignment.
Build. Add integration, load, restart and security tests. Use synthetic contacts and harmless operations. Test delayed events, expired credentials, provider outages, duplicate messages, deleted content and actions with unknown outcomes.
Check. Each test shows the customer-visible result and the downstream state, including a safe stop.
Ask AI to apply these decisions
Templates and examples
Every file uses synthetic data. Adapt it to your own policies and test it somewhere safe before it touches customers.
- evaluation-cases.jsonJSON fixture or schema
- experiment-plan.mdTemplate (Markdown or text)
- selection-cases.jsonJSON fixture or schema
Keep it working
Launch is the start of the work, not the end. These are the recurring jobs this part of the agent needs, and who usually owns them.
| Cadence | Owner | Work |
|---|---|---|
| Every change | Engineering | Turn each confirmed bug into a focused regression test. |
| Monthly | Quality owner | Refresh a controlled sample as products and customer behavior change, and keep a stable set for comparison. |
| Monthly | Quality owner | Review calibration and uncertain cases. |
| During experiment | Quality owner | Watch guardrails while the test runs. Analyze segments and follow-up outcomes before rollout. |
| On change | Engineering | Rerun affected cases after infrastructure or API changes. |
Test your decision
A candidate scores better, but one arm used a fallback model for many requests. What should the report say?
Show the expected decision
It should state the intended and actual model routes and what kind of experiment this was. For a pure model comparison, the substituted runs are invalid, so the claim does not hold. For a comparison of the whole service, the runs can stay in, provided the report shows how often the fallback was used and analyzes results by assigned arm.
