Measure AI support resolution, CSAT and service quality
The short answer
Measure the work your agent actually delivered and the outcomes you can confirm once each conversation has had time to settle. Keep involvement, containment, resolution and satisfaction as separate numbers. Then link every conversation to its configuration, sources, tool results and delivered reply, so your team can explain any score and know what to fix.
Keep a decision record for every reply
You can’t trust a score until you can reconstruct what happened in a single conversation. Each turn (one customer message in, one agent reply out) touches several systems: the helpdesk, your application, the search index, the model provider and any tools the agent called. Give every turn a correlation ID, a shared identifier written into each system’s logs, so you can join those records back together when something goes wrong.
Against that ID, store the helpdesk event, the conversation version, the configuration the agent selected, the IDs of the sources it retrieved, each tool outcome, the model requests and the reply that was delivered. Add timings, error categories and the provider route the request actually took, which can differ from the one you configured if a gateway rerouted it. Keep payload detail to the minimum a stated investigation needs. You rarely need the full request body, and every extra copy of customer text is another store you have to secure and delete.
Test the record from your support lead’s side. Starting with nothing but a ticket ID, they should be able to answer four questions: what information was available to the agent, which rule or procedure applied, which actions were confirmed, and what actually reached the customer. Show the recorded evidence for each answer. If you also generate an explanation of why the agent replied the way it did, label it as an explanation. It’s a fresh interpretation produced after the fact, not a record of the model’s private reasoning.
Give every metric an explicit denominator
Most disagreements about AI support numbers are really disagreements about denominators, the set of conversations a rate is calculated over. “80% resolution” means little until you know 80% of what: all incoming traffic, every conversation the AI replied to, or only the ones old enough for the outcome to be known. Write each definition into a metric dictionary and keep it stable across releases, so a change in the number reflects a change in the service rather than a change in the math.
| Metric | Definition for this guide | Main trap |
|---|---|---|
| AI involvement | Conversations with a delivered AI reply / eligible conversations | Counting replies that were generated but never delivered |
| Operational containment | Mature AI-involved conversations with no human handling / mature AI-involved conversations | Treating customer silence as success |
| Verified AI resolution | Mature AI-involved conversations with positive resolution evidence and no qualifying reopen / mature AI-involved conversations | Counting an automatic close as evidence |
| Ticket deflection | Estimated reduction in tickets against a valid baseline or control group | Claiming cause and effect from contained tickets alone |
| Customer CSAT | Ratings submitted by customers, reported with the response rate | Ignoring which customers chose to answer |
| AI-estimated CSAT | A model-based satisfaction estimate, reported with coverage and calibration | Presenting a prediction as a customer rating |
A few terms in the table need pinning down. An eligible conversation is one your launch scope says the agent should handle, by channel, language and intent. A mature conversation is one that has passed your agreed observation window, the period after the last message in which a reopen or follow-up would normally show up. A qualifying reopen is one your policy counts against the AI, such as the same customer coming back about the same issue inside that window. Measure outcomes before the window closes and you’ll count customers who simply haven’t come back yet as successes.
The three headline metrics describe different events. Containment tells you the conversation stayed with the AI and no human stepped in. Resolution tells you the customer’s problem actually ended. Deflection estimates tickets that never opened at all, which you can’t count directly, so it always depends on a baseline. A frustrated customer who gives up counts as contained, which is why containment alone can’t stand in for resolution.
A worked metric example
The numbers here are made-up arithmetic, not a My AskAI result. Suppose 100 conversations are eligible and 80 of them receive a delivered AI reply, so AI involvement is 80%. Only 60 of those 80 have reached the end of the observation window. Of those 60 mature conversations, 42 had no human handling, so operational containment is 70%.
Now look for evidence. If 30 of the 60 mature conversations have positive resolution evidence and no qualifying reopen, verified AI resolution is 50%. The other 30 each need an explicit outcome recorded as unresolved or unknown rather than being left blank. The remaining 20 AI-involved conversations are still too recent to judge. Report them as pending, and don’t fold them into a mature-outcome denominator, where they would quietly inflate or deflate the rate.
| Step | Count | Rate |
|---|---|---|
| Eligible conversations | 100 | – |
| Delivered AI reply (AI involvement) | 80 | 80% of eligible |
| Mature AI-involved conversations | 60 | 20 still pending |
| No human handling (operational containment) | 42 | 70% of mature |
| Positive evidence, no reopen (verified resolution) | 30 | 50% of mature |
Notice the 20-point gap between containment and verified resolution. Those 12 conversations stayed with the AI but have no evidence the customer was helped, and they’re the first ones worth reading. Neither number tells you anything about ticket deflection, which needs a credible baseline or control group and attention to changes in overall traffic. If your own policy uses different denominators, that’s fine. State them clearly and keep them stable from one version to the next.
Score conversations once the outcome is visible
Many quality signals come from automated scoring. A separate model, often called a judge, reads the conversation and grades it against a written rubric: was the answer correct, was it grounded in an approved source, did it resolve the request. Score a conversation only once its outcome can be observed, either after it escalates or after the agreed delay. Give the judge the information that was available at the scoring cutoff, and store the event time separately from the scoring time so a late score isn’t credited to the wrong week or release.
When you compare two versions of your agent, score both with the same independent judge model and rubric. If the judge changes at the same time as the agent, you can’t tell which change moved the score. The same applies to AI-estimated CSAT. Calibrate it by checking the estimate against real customer ratings on conversations where you have both, so you know how far to trust it before anyone reads it as a satisfaction score.
Not every conversation will get a score, and the reasons matter. Store them separately: ineligible, missing context, parser failure (the judge’s output couldn’t be read) and judge failure (the scoring call errored or timed out). Show coverage, the share of eligible conversations that received a score, next to every score on your dashboard. Labeling every unscored conversation “not applicable” quietly removes the hardest cases from the average. If you change the judge, give it a visible version number and plan how you’ll compare old and new scores before you rescore anything.
Turn inspection into a specific fix
A wrong or missing answer can come from several places: the source was wrong, retrieval missed the right source, the agent picked the wrong task, a tool failed, or the reply never arrived. Each of those has a different owner and a different fix. A generic prompt edit won’t repair a failed delivery or a stale search index, and it can make other answers worse.
| What the record shows | Likely cause | Where the fix goes |
|---|---|---|
| The right article exists but wasn’t retrieved | Retrieval or indexing | Search configuration, chunking or index freshness |
| The retrieved article says the wrong thing | Source content | The policy or content owner |
| The right source was used but a rule was ignored | Instructions or task selection | Guidance or task definition, with new tests |
| A tool returned an error or bad data | Integration | The API or adapter owner |
| A reply was generated but no message reached the customer | Delivery | The helpdesk integration owner |
My AskAI’s public Inspect & Logs and Echo documentation shows one product-level example of this workflow: start from a conversation and ask which sources, guidance and tools shaped the reply. If you build a similar internal assistant, scope its access to logs by permission and have it cite the evidence it relies on. Keep any edits it proposes inside your normal controlled release workflow rather than letting it change the live agent directly.
Alert on action, not just activity
A useful alert tells someone what to do. It names the condition, the affected cases, the owner and the next step. “Error rate high” sends someone digging through dashboards. “Replies to the helpdesk are failing to send, here are the affected conversations, integrations on-call owns it, and this is the runbook for reconciling pending deliveries” gets the problem fixed.
Watch the signals that point to customer harm: queue age (how long accepted messages have been waiting), delivery failures, failed handovers, writes with an unknown outcome, source freshness and unusual spend. Review quality trends by intent, language and channel as well as overall, so a healthy average can’t hide a segment that’s failing. A strong English score can sit on top of a broken Spanish refund flow.
Set thresholds from your own risk and baseline traffic rather than copying another company’s numbers. Then test the whole path: trigger a synthetic alert, confirm it reaches the right owner, and check that the linked runbook still works. Logging more fields won’t help if nobody can find the conversation that matters when the alert fires.
Ask AI to explain this guide
Build decisions and checks
Use these decisions to turn the worked case into your own implementation. A check is complete only when you have evidence for the stated result.
Connect one conversation to its evidence
Required. When helpdesk, application and model logs live apart, a failure is hard to piece back together.
Build. Carry one correlation ID through receipt, worker, retrieval, model, tool and delivery. Store versions, timings, decisions, sources, error types and action receipts. Collect only the payload detail you need, within the access policy.
Check. Starting from a helpdesk conversation ID, find the relevant evidence without a secret appearing in any URL or log.
Show decisions in plain language
Required. A teammate needs to know what happened, not read a made-up explanation of the model's hidden reasoning.
Build. Show a factual timeline: event accepted, rule applied, source used, tool result, delivery and transfer. Link each step to its evidence. Keep model output distinct from confirmed system actions. Give support a readable view and engineers deeper diagnostics.
Check. A reviewer can pinpoint the failed step in a test incident. Explanations without evidence are marked unknown.
Define involvement, resolution and satisfaction
Required. A closed ticket or a customer who never replied isn't proof the problem was solved.
Build. Write a metric dictionary with denominators, exclusions and observation windows. Keep eligible traffic, AI involvement, operational containment, verified resolution, human CSAT (customer satisfaction) and AI-estimated CSAT as separate measures. Track reopened cases and transfers. Don't count automated closure as evidence of resolution. Record why a conversation can't be scored, and keep missing data, parser failures and ineligible cases as separate reasons.
Check. Recalculate a small sample from raw events. Missing scores stay missing, and outcomes that haven't arrived yet don't turn into false successes.
Make alerts actionable
Required. An average quality score can hide a broken tool or a regression in one language.
Build. Alert on unsafe actions, failed handovers, delivery errors, queue age, knowledge freshness, tool failures, latency and cost. Give each alert a severity, an owner and response steps. Alert on sustained rates where that fits, and immediately on critical events.
Check. Send a synthetic alert to the responsible team and follow the runbook through to a confirmed recovery.
Ask AI to apply these decisions
Templates and examples
Every file uses synthetic data. Adapt it to your own policies and test it somewhere safe before it touches customers.
- metric-dictionary.csvSpreadsheet worksheet (CSV)
- metric_example.pyRunnable Python example (synthetic data, no network calls)
- decision-record.jsonJSON fixture or schema
metric_example.py"""Synthetic denominator example. Run: python3 metric_example.py.
These are teaching values, not My AskAI performance results.
"""
def rate(numerator, denominator):
if denominator == 0:
return None
if numerator < 0 or denominator < 0 or numerator > denominator:
raise ValueError('inconsistent counts')
return numerator / denominatorKeep it working
Launch is the start of the work, not the end. These are the recurring jobs this part of the agent needs, and who usually owns them.
| Cadence | Owner | Work |
|---|---|---|
| Weekly | Engineering | Check for missing trace spans, retention, trace cost and sensitive data. |
| Weekly | Support lead | Review tickets that were hard to diagnose and improve the evidence shown. |
| Weekly | Quality owner | Review metrics by topic, language, channel and identity state. |
| Weekly | Service owner | Review noisy alerts and incidents the alerts missed. |
Test your decision
A dashboard shows a higher resolution rate right after the team changed the judge model. Can the team claim the agent improved?
Show the expected decision
Not from that result alone, because two things changed at once. Score a shared bridge sample under both judge versions while keeping the agent configuration fixed, and check that coverage didn’t shift. If the new judge simply grades more generously, the apparent gain belongs to the judge, not the agent.
