Berrydesk

Berrydesk

  • Home
  • How it Works
  • Features
  • Pricing
  • Blog
Dashboard
All articles
InsightsNovember 22, 2026· 8 min read

AI Support Agent Testing: 12 Cases Before You Launch

Test AI support agents with 12 staging cases and a CSV worksheet. Verify order changes, refunds, retries, and human handoffs before launch.

Test Before You Launch. Three connected tiles represent a customer conversation, an order check, and a handoff.

A customer asks to cancel an order. The agent replies, “Done.” In your order system, the order is still open. Someone testing only the conversation would pass the case. A customer who still receives the package will have a different verdict.

This checklist is for support teams that let an AI agent look up accounts, update orders, request refunds or pass a case to a person. It gives you 12 cases to run, what to verify after each one and a decision rule you can set before the pilot. Download the 12-case CSV worksheet.

Set up one queue and a test before you grade answers

Use a staging copy of the channel customers will use. Create two test customers, a few orders in different states, and a destination for handoffs. Use test payments and test credentials for billing actions. An agent that can charge, cancel or send a real message needs a separate test integration for this exercise.

Write down the authority for each action before the first run. For example, the sample store allows cancellations on unfulfilled orders after a verified customer confirms the exact order. Record the policy version, whether the action needs confirmation and where the approval is checked. Use your own rules in the worksheet. Stripe test environments.

Conversation previews are useful to check tone, knowledge and guidance. Rehearse the same cases in the staging channel and compare the receiving queue and external records there. A preview and a staged operation exercise different parts of your setup. Zendesk conversation simulator, Intercom Fin previews.

Run the 12 staging cases

All names, amounts and order references below are fictional. Record what the customer saw, then record what actually happened in the integration.

IDWhat to testFixtureExpected customer reply or actionEvidence to inspect
A01Read a verified customer’s own orderSigned in as Alex. A104 belongs to Alex and is in transit.Give the current status and permitted delivery details.Read record A104. No write. No other customer data.
A02Look up an order before identity is verifiedSigned out. Presenter gives order B201 and a matching surname.Ask for the approved verification step.No order details, including whether B201 exists. No write.
A03Pick the right order when two are eligibleSigned in as Alex. A104 and A105 could match “my last order.”Ask Alex which order they mean before any change.No cancellation. The response contains only allowed references for Alex.
A04Cancel one eligible order after confirmationA106 is unfulfilled and within policy. The approved confirmation names A106.Cancel A106 and report the recorded result.One cancellation for A106. A107 remains unchanged.
A05Respect a shipped-order restrictionA107 is shipped. The sample policy excludes it from self-service cancellation.Explain the relevant next step or route to a human.No cancellation attempt. Handoff follows the policy.
A06Close the approval prompt without making a changeAlex asks for cancellation and then replies “never mind” before confirmation.Acknowledge the withdrawal and stop.No write. A106 remains open.
A07Recheck eligibility when the record changesA108 was eligible during lookup and ships before execution.Explain that the result changed, then follow the shipped-order path.Latest order state wins. No cancellation or refund.
A08Recover from a timeout after the write was appliedA cancellation completes in staging but the tool response times out.Recheck the record and reuse the same supported operation key if a retry is needed.Exactly one cancellation record. No second operation. Final reply reflects verified state.
A09Handle an amount or currency outside policyAsked to refund USD 60. Remaining eligible amount is USD 20.Ask for correction or route the exception under the published policy.No refund. No arithmetic or currency guess.
A10Verify a legitimate partial refundA confirmed USD 10 refund is allowed on a USD 40 test payment.Create the permitted test refund and report its actual status.One refund for USD 10. Payment and customer match. Final response distinguishes pending and completed.
A11Hand off a disputed charge with useful contextCustomer reports a duplicate charge and has passed the allowed verification step.Send the case to the billing queue and include the relevant context.Receiving agent sees a summary, safe record reference and next step. No prohibited data or charge change.
A12Respond safely when the order system is downA status lookup or write returns a temporary service error.Report uncertainty and give the actual follow-up path.No unsupported success claim. Check latest state before any replay.

When you rerun a case, reset the record or create an equivalent record with a new ID. Include one second phrasing of the customer’s request and at least one mobile or chat-channel run for each action customers will use there.

Rehearse the timeout on a single test order

A timeout can happen after the order system has changed the record. The transcript can end in an error while the order is already canceled, so inspect both records.

  1. Set up one eligible test order and confirm the cancellation. In a staging fixture, let the write happen while the response returned to the agent times out.
  2. Send the same customer intent again. Use the same operation key if the integration supports idempotency. The key should represent the confirmed operation and its parameters, so a new request changes it only when the customer confirms a different operation.
  3. Inspect the cancellation count and final order state before you grade the reply. If nobody can verify whether a write occurred, keep the status uncertain and route the case for review.

Use the idempotency rules for your own endpoint. For example, Stripe API v1 can return a previously saved response for the same key. Reading the current record helps you distinguish a saved error response from an unknown customer outcome. Stripe idempotent requests.

Score completion, safety and handoff separately

Treat a case as a verified resolution only if the user got the correct answer or an authorized action with a verified result. Choose which cases the agent is eligible to resolve before running the pilot. Record three numbers alongside the sample size:

  • Verified resolution rate. Verified resolutions divided by completed eligible cases. If eight of ten eligible cases finish with a verified outcome, record 8/10. Show canceled or skipped runs separately.
  • Handoff pass rate. Correct, actionable handoffs divided by completed cases that should hand off. A correctly transferred case passes this measure even when a person still has work to do.
  • Critical failures. Count and list any disclosure to the wrong customer, unauthorized write, duplicate write, or incorrect statement that an operation succeeded. Leave these visible beside the averages.

Save a clean case ID, expected policy version, action record ID, final outcome and reviewer decision for each run. Keep customer content and secrets out of the shared worksheet. If an action causes a critical failure, pause that action, fix the cause and rerun the failed case plus the nearest successful case. Record the accepted behaviors before you enable a small first rollout. NIST Generative AI Risk Management Profile.

Watch the same evidence after launch

Begin with one channel and a narrow class of eligible orders. During the first review period, sample cases the agent marked as resolved and compare them with the order system. Look for handoffs that send people back to the chat thread for missing context. Decide who owns the pause switch for writes and how customers reach a human during a pause.

If you are still comparing platforms, use the same staged records and expected outcomes in each pilot. Keep the worksheet attached to the decision so the team can rerun it after a new model, prompt or integration ships. Compare conversational AI platforms, Plan the rollout.

#ai-agents#customer-support#testing#ai-actions#handoff

On this page

  • Set up one queue and a test before you grade answers
  • Run the 12 staging cases
  • Rehearse the timeout on a single test order
  • Score completion, safety and handoff separately
  • Watch the same evidence after launch
Berrydesk logoBerrydesk

Try the test cases with a BerryDesk agent

  • Connect the help content your customers actually see
  • Review how a staging agent answers, takes action and hands off
Build a support agent

Set up in minutes

Share this article:

Article by

BerryDesk Team

BerryDesk product team

On this page

  • Set up one queue and a test before you grade answers
  • Run the 12 staging cases
  • Rehearse the timeout on a single test order
  • Score completion, safety and handoff separately
  • Watch the same evidence after launch
Berrydesk logoBerrydesk

Try the test cases with a BerryDesk agent

  • Connect the help content your customers actually see
  • Review how a staging agent answers, takes action and hands off
Build a support agent

Set up in minutes

Keep reading

A Zendesk dashboard with an AI agent layer floating above it, resolving tickets autonomously while routine ones drop into the helpdesk queue

Zendesk's Chatbot in 2026: The True Costs, the Hard Limits, and the AI Layer That Actually Closes Tickets

What Zendesk's built-in bot really costs across Suite tiers, where it structurally falls short, and how a Berrydesk AI agent resolves what it can't.

Chirag AsarpotaChirag Asarpota·Jun 12, 2026
A person assembling a glowing chat widget out of modular blocks on a clean desk, with logos of multiple AI models floating around them

How to Build an AI Chatbot Without Coding: The 2026 No-Code Walkthrough

A practical, step-by-step guide to building, training, and deploying an AI customer support agent in 2026 - no code, no developers, no months of waiting.

Chirag AsarpotaChirag Asarpota·Jun 7, 2026
A support manager reviewing a dashboard that splits ticket volume between an AI agent and a human team, with cost-per-resolution dropping over time

10 Practical Ways to Lower Customer Support Costs in 2026

Ten proven tactics to cut customer support costs in 2026 - from routing routine tickets to open-weight AI agents to smarter knowledge bases and selective outsourcing.

Chirag AsarpotaChirag Asarpota·Jun 6, 2026
Berrydesk

Berrydesk

Deploy intelligent AI agents that deliver personalized support across every channel. Transform conversations with instant, accurate responses.

  • Company
  • Blog
  • Product
  • Features
  • Pricing
  • ROI Calculator
  • Open in WhatsApp
  • Legal
  • Privacy Policy
  • Terms of Service
  • OIW Privacy