Test an AI Assistant Before Launch With Repeatable Questions

Build a test set from real support questions

Testing before launch is most useful when the questions reflect the work customers actually bring to the team. Start with a fixed set drawn from common support themes, then add cases that reveal where the assistant should pause. A small deliberate sample is easier to review than a large collection with no expected outcomes.

Include four kinds of prompt. A common question checks whether the approved source supports an ordinary answer. An ambiguous question checks whether the assistant asks for the missing detail instead of guessing. A missing-source question checks whether it acknowledges that the material does not establish an answer. A handoff question checks whether it recognizes a decision that belongs to a person. Write expected behavior beside each exact prompt before testing.

For example, a service business might test a published appointment requirement, an incomplete question about which location is open, a request for a policy detail absent from its documents, and a request to approve an exception. The correct outcome is not always a direct answer. A clarification or an appropriate referral can be the strongest result when the evidence or authority is missing.

Prepare sources and expected outcomes

Before using the assistant test page, confirm that the relevant source is authoritative and current. Keep the test scope aligned with the material supplied to the assistant. If the source says only that appointments are required, do not expect a precise opening time that appears nowhere in the approved material. Record the source and the specific fact that supports each expected answer.

Resonoon’s assistant editor includes a test page, and its knowledge editor supports PDF and Markdown uploads. The test page is for checking responses; the team should retain its own repeatable question set and score record. Keep the rubric and results in a shared team document so reviewers can apply the same criteria on each run.

Agree on a compact scoring rubric. One useful approach marks each response as correct and complete, useful but requiring a person, unsupported or incorrect, or unclear. Add a short note about the evidence and the next action. Avoid scores such as “good” without criteria. Reviewers should be able to explain why a response passed and what source or instruction they would change after a failure.

Run tests consistently

Use the same wording when comparing revisions. Record the assistant version or the configuration changes being evaluated, the prompt, the answer, the expected result, and the reviewer’s assessment in a team document. Use a consistent record format so another reviewer can repeat the comparison and identify the configuration tested. Keep personal or sensitive customer data out of test prompts unless it is necessary and approved for the test.

Test paraphrases separately from canonical questions. A customer may describe a product in their own words or omit a key condition. Also test a prompt that sounds confident but asks for something unsupported. Do not lead the assistant with the expected answer or repair the question after a disappointing result; doing so weakens the test.

When an answer fails, classify the likely cause before editing. A wrong policy detail may indicate a stale or conflicting source. A missing boundary may call for clearer role guidance. An escalation with no owner points to an operating gap. Change one cause and rerun the identical prompt plus its related cases. This makes improvement attributable and helps catch regressions in nearby questions.

Make launch readiness a team decision

Set a launch threshold before seeing the results. For example, require every policy-critical question to be either accurate or safely handed off, while tracking less consequential clarity issues separately. The exact threshold depends on the business and should be approved by the people responsible for customer support and policy. A favorable average must not hide a critical wrong answer.

Before launch, confirm the human fallback, the intended channel, and the source owner. If the assistant is used through the web widget, review the customer-facing page after configuration and ensure the team knows how to respond to a handoff. Do not infer staffing coverage or response times from a successful test.

Retain the test set and rerun it after meaningful changes to role guidance or source material. Resonoon’s AI chatbot widget provides context for the website experience. A test set cannot prove that every future conversation will succeed, but it can reveal known weaknesses before customers encounter them and make later changes easier to assess.

Name a reviewer responsible for signing off on policy-critical cases. If reviewers cannot agree on the expected result, resolve the policy question before marking the test as passed.