Launch an AI support agent with evidence and rollback

The support case

Define readiness before the last test

If you decide what “ready” means after the final test run, the definition tends to bend to fit the results. Write it down first. Set the supported intents (the kinds of request the agent will handle, such as order status or address changes), channels, languages and the limits on any actions it can take. Agree the critical pass/fail rules, plus realistic latency and cost targets taken from your current baseline. Name the person who decides whether the first group of customers can start, and the people who are allowed to pause the agent.

Then treat the launch checklist as an evidence record rather than a list of ticks. Every checked item should point to a test result, a source or an accountable decision. “Webhook works” isn’t enough on its own. For the intended workspace, the record should show that the integration verifies the helpdesk’s signature on each event, keeps accepted events safe through a restart, doesn’t produce two replies when the same event arrives twice, and reads the posted message back from the helpdesk to confirm delivery.

Expand exposure in controlled steps

Each stage of a rollout proves something specific and leaves something else unproven. Notes mode (sometimes called draft or copilot mode) posts the agent’s reply as an internal note for a teammate to review and send, so your team sees real answers on real tickets before any customer does. A limited live cohort is a small, defined slice of eligible traffic, such as one channel or one intent, that receives automated replies while everyone else is served as before.

Move to the next stage only when enough conversations in the current one have mature outcomes, meaning they’ve passed your observation window, and those outcomes meet the release rules you set in advance. In notes or shadow mode, keep evaluation separate from real actions. A draft that says “I’ve processed your refund” must not have actually triggered a refund that a teammate never approved.

Make the human path work first

Handover is the safety net for everything the agent can’t or shouldn’t do, so prove it works before you rely on it. Confirm the receiving queue, the permissions, the business hours and the fallback destination when that queue is closed. Then have a support teammate open a transferred test case and carry on the conversation. Check that the handover summary contains the facts they need and the actual outcomes of any tools the agent called, not just a restatement of the customer’s first message.

Test the case where no human is available, too. The customer needs accurate wording and a real next step. Telling someone they’re being connected to a person is misleading if the system only creates a ticket for review on Monday morning. Decide who handles urgent cases outside normal hours, and make sure the agent recognizes where its role ends, for example by handing over a safety complaint instead of trying to resolve it.

Check the running release

A successful build log tells you code was packaged. It doesn’t tell you what’s running. A production service can still be using an old configuration, or routing requests to a different model than the one you tested. Record the application version and the effective model, prompt, task, tool and knowledge versions, meaning the versions the live service actually loaded, not the ones you intended to ship.

After each deployment, run a smoke test: a single synthetic or otherwise harmless request sent through the intended environment from end to end. Read back the resulting helpdesk message, the assignment and any permitted downstream state, such as a harmless tool result. Keep development credentials and customer fixtures out of production tests. The smoke test should be narrow, authorized for that environment and come with a clear rule for cleaning up whatever it creates.

Practice pause and rollback

Pause and rollback do different jobs. A pause stops the agent sending new public replies and making new business changes, such as refunds or cancellations, while keeping the receipts and reconciliation work it needs to finish safely. Test a pause while a model call is still in progress, and check that the delayed reply can’t slip out after the pause takes effect. It also helps to be able to disable a single tool without stopping all automated support, so a failing refund integration doesn’t take order-status answers down with it.

A rollback restores the previous compatible configuration. It needs that configuration to still exist and a tested way to switch back to it. It doesn’t reverse refunds already issued or messages already sent. Keep those completed actions in the record and reconcile any whose outcome is unknown. If the release includes a change to how sources are stored or indexed, test either its rollback or a safe way to recover forward as well.

Run a short exercise with support and engineering together before launch. Your support team should know where the pause control is, who may use it, and what customers will see while the agent is paused. If only engineers know how to stop the agent, the people watching the conversations can’t act on what they see.

Observe before expanding

During the first cohort, watch for delivery failures, unexpected handovers, customers repeating the same question, stale jobs that sit in the queue, and tool calls that were denied. Compare outcomes against your baseline only after the observation window is complete, because early numbers overstate success while unhappy customers have yet to come back. Keep a short daily decision record during the initial rollout: what you saw, what you changed, and whether the cohort continues, pauses or grows.

When you expand, change one meaningful dimension at a time, such as adding a new intent or a new channel. A working English order-status agent doesn’t show that you’re ready for multilingual refunds, which bring new languages, new policy and a write action all at once. Each new boundary needs its own tests and its own owners.

Keep it working

Launch is the start of the work, not the end. These are the recurring jobs this part of the agent needs, and who usually owns them.