Direct answer: Give the chatbot real customer questions, an approved answer source, and a pass condition for each. Test questions it should answer, questions it cannot answer from the supplied material, ambiguous questions, and cases that need a person. Launch only after the business has reviewed both its answers and its handoff path. A fluent demo is not an acceptance test.
Two 2025 research papers explain why. Peng and colleagues' ACL study evaluates retrieval-augmented systems on unanswerable requests across four datasets; no tested component and prompting configuration was consistently best for both answerable and unanswerable queries. Zhou and colleagues' COLING study separates an answer's correctness from whether its cited material actually supports it, finding vulnerability when retrieved knowledge is partial or flawed. These are research settings, not measured results for a specific retailer's support desk. Together they justify testing answer, evidence, and refusal separately.
Build four lanes from actual support work
| Lane | Example on a commerce or service site | Pass condition |
|---|---|---|
| Answerable | “How long is delivery to this postcode?” when the current policy states a rule | Gives the current rule, relevant exception, and a link to the maintained policy. |
| Unanswerable | “Will my custom order arrive by Friday?” when no production date is available | Says it cannot verify that date and offers a human or order-status route; does not invent a promise. |
| Ambiguous | “Can I change it?” without an order or booking context | Asks what “it” refers to before stating a policy. |
| Human-required | Complaint, refund exception, account-specific issue, or a high-consequence claim | Explains the handoff and sends the available context to the right team with appropriate customer consent. |
Start with a manageable sample from support tickets, site search, and sales questions. Include wording customers actually use, not only polished FAQ headings. For each case, write the approved source URL or internal policy version, expected answer elements, prohibited claims, and escalation owner. The business must approve the policy; a website supplier cannot infer a client's refund, shipping, medical, or account rules from a model response.
Score failures that matter to the customer
Review each response for three things: Is the stated fact correct? Does the cited or retrieved material support that exact fact? Did the bot refuse or escalate when it lacked support? Log failures by category and retest after changes to model, retrieval, prompt, or policy. Avoid using “percentage of chats without a human” as the sole success measure; an ungrounded answer can suppress a ticket while creating a worse customer problem. The research above supports this separation of correctness and groundedness, not a specific pass-rate threshold.
The handoff is part of the product. Confirm that the contact route works, the customer knows when to expect a reply, and the human can see the relevant question without forcing the customer to repeat it. The contact-page routing guide covers the downstream path. If customer data enters a third-party chatbot, scope retention, permissions, and disclosure with the provider and the business before launch; this checklist does not certify legal compliance.
This article concerns support on a website after a customer asks a business question. It differs from choosing an AI guide for a live event or writing general self-service content. An ARCHV digital-product or website engagement can define the test set, UI states, and human handoff with the client, then retest as the underlying policies change.