How to Test a Chatbot: AI Support Agent vs FAQ Bot

At Zealoop’s Free plan limit of 50 conversations per month, the agent stops answering, sends a holding reply, and adds the conversation to the inbox. That is a support-bot behavior worth testing just as deliberately as an answer about a product feature. Knowing how to test a chatbot means checking the behavior its architecture and operating limits actually require.

A scripted FAQ bot should pass predictable checks for intents, branches, and fixed responses. An AI support agent needs those checks plus tests for grounded answers, paraphrases, customer-data permissions, guarded actions, and safe fallback. Teams should automate repeatable pass/fail cases and use exploratory sessions to find gaps those cases do not cover.

The Zealoop conversation limits above are plan capacities, not the price of paid plans. Zealoop counts one conversation as one thread with one visitor, regardless of message count. (Plan details)

How to test a chatbot across both bot types

Chatbot testing checks whether a bot produces the expected answer or action, follows the correct conversation path, respects safety boundaries, and gives customers a usable experience. The test oracle differs by bot type: a rule bot can often be judged against a specific branch, while an AI support agent needs criteria for meaning, source support, uncertainty, and actions as well as its visible reply.

Set outcomes before running cases. A support test passes when the bot does the right thing, not merely when its response sounds fluent. For each scenario, define:

• The customer’s goal and relevant account state.
• The facts or system result required for a pass.
• The route, permission decision, or state change expected.
• The safe fallback if the answer or action cannot be completed.

For a documentation question, that could mean requiring the correct setup steps and their source. For an account request, it could mean returning the authorized customer’s status without exposing another person’s data. The same evaluation should not treat a safe refusal as a failed answer or a confident unsupported claim as a success.

Build test cases around customer intent and wording

Start with real support questions and group them by intent: setup, password reset, billing, account lookup, or a request to change a subscription. Include both frequent cases and lower-frequency cases with higher consequences, such as a request for private data or an action with billing impact. Preserve the wording customers use rather than rewriting every message as a clean FAQ.

For one intent—checking whether a feature is included—test expressions such as “Does my plan have exports?”, “can I export on this plan,” and “where do I download a CSV?” Define the shared expected facts and note when the final phrasing needs a clarification, such as which plan the customer has. The bot should not invent a plan or treat every paraphrase as a different policy.

Give each test case a compact record: input, intent, expected answer or action, source or account fixture, risk, actual result, and pass/fail reason. Add ambiguous prompts and unsupported questions. “Is that included?” should prompt for the missing product or plan detail when context does not resolve it; it should not trigger a guess. For supported knowledge questions, Zealoop says it cites the source section for each answer and validates grounding before sending, abstaining when validation fails. Those are observable behaviors to include in the expected result. (Zealoop)

Test dialog logic, routes, and conversation state

A correct answer can still fail if the bot reaches it through a broken path. Test the opening message, intent recognition, routing, follow-up questions, context retention, and completion or recovery as one flow. For example, after a customer says a password reset did not work, check that the next turn responds to that failure rather than repeating the same instructions.

For each important route, vary the initial utterance and inspect both the conversation and available state evidence: selected intent, captured fields, queue, tool call, and final status. A standalone bot should be tested through its own conversation path. If a separate orchestrator directs messages to bots or services, test the transitions between components as well as each individual flow.

An illustrative release exercise might contain 24 scripted cases: eight common intents, eight paraphrases or incomplete prompts, and eight routing or recovery paths. A reviewer could then spend a separate 30-minute exploratory session following unexpected replies and changing topics. If the scripts expose two missed routes and the session uncovers three unlisted context failures, that gives the team five defects to triage and add to its test set. These are example results, not a benchmark; the point is to record what each method found and turn reproducible failures into regression cases.

Check answer grounding and unsupported questions

For each knowledge answer, list the required facts, important conditions, prohibited claims, and the approved source. Then compare the response with those criteria. If a customer asks about a setup requirement and a compatibility limit, an answer covering only setup is incomplete even if that part is correct. If a statement is absent from the approved documentation, fluency does not make it grounded.

Use meaning-based acceptance criteria rather than requiring identical wording. A response may pass with different phrasing if it preserves the required facts and conditions. It should fail if it omits a critical limit, combines conflicting instructions, or asserts a detail that the source does not support. Test unsupported questions separately, with an expected clarification, abstention, or escalation path.

Knowledge changes should trigger focused retesting. When a source page changes, rerun cases that depend on it and relevant neighboring topics, not just the one question used to confirm the edit. Teams can organize their approved sources and coverage using the Zealoop knowledge documentation.

Verify permissions, guarded actions, and human handoff

Treat information retrieval and account changes as separate test surfaces. A bot may be allowed to explain how cancellation works but still need authorization or confirmation before changing an account. Use test identities and controlled records; verify both the customer-facing reply and the resulting system state.

Include cases for an allowed lookup, a denied lookup, an action that needs confirmation, and an unauthorized action. A denied request should leave the protected record unchanged. A confirmation-required action should not proceed until the expected confirmation is received. For a support agent connected to customer data, test a request for another customer’s information and make sure the response does not reveal it.

Test escalation as a completed route rather than a visible button. When the bot cannot resolve a case, check that the visitor gets a clear next step and that the receiving human can see useful context: the original issue, information already requested, and any relevant lookup or attempted action. For Zealoop’s conversation cap, a boundary test should verify the holding reply, inbox addition, and account email at the limit; it should also check that several messages in one visitor thread count as one conversation, not several. (Zealoop plan details)

Measure experience and choose regression tools

Functional correctness is necessary but does not cover usability or operations. Record response time, availability, clarity, consistency, errors, and whether the customer can identify the next step. For every failure, save the input, transcript, expected and actual outcome, source or account fixture, route and state, action result, timing, and escalation outcome. This evidence makes a failure reproducible and helps locate whether it came from content, routing, permissions, or infrastructure.

Use scripted checks for known routes, required facts, permissions, and limit behavior. They are repeatable and useful after a change, but only cover the cases written down. Use human exploratory testing for mixed intents, unclear phrasing, surprising follow-ups, and confusing recovery. It can expose unknown gaps, but a finding needs a saved transcript and expected result before it becomes a reliable regression check.

Botium and DeepEval are names teams may encounter when comparing chatbot testing tools. The choice should follow a proof-of-concept against the team’s actual checks: can the selected setup reproduce the conversation, assert the needed outcome, and preserve evidence when a case fails? For a small SaaS team, begin with its existing test runner for deterministic checks and reserve reviewer time for risk-heavy exploratory conversations; add a specialized tool when a demonstrated coverage or reporting gap justifies it.

Run a repeatable test cycle after each change

A useful cycle is short enough to repeat but broad enough to cover the risks of an AI support agent:

1. Prioritize common intents and high-impact cases involving billing, private data, or actions.
2. Write expected answers, sources, routes, permissions, state changes, and fallback behavior.
3. Run scripted functional, grounding, and safety checks.
4. Explore ambiguous and multi-turn conversations that are not fully represented in scripts.
5. Triage failures, fix the cause, and rerun affected cases.
6. Add each confirmed defect as a regression case.

Repeat the relevant checks after changes to prompts, knowledge sources, integrations, routing, or action permissions. A change to an answer source may require grounding tests; a permission change calls for access-boundary and state checks. Keeping cases linked to the behavior they cover makes retesting more targeted than rerunning a demo and calling it ready.

Which should you choose?

For a simple FAQ bot with stable menus, prioritize scripted intent, branch, and recovery checks. For an AI support agent that answers from documentation, add source and unsupported-question tests. If it reads customer records or can take actions, add explicit authorization, confirmation, state-change, and handoff checks before launch. Use exploratory sessions whenever a workflow is new or the likely customer wording is not yet represented in the scripts.

The practical choice is not one method or the other: scripted regression is the release baseline, while exploration expands that baseline. A team can start with the highest-risk flows and grow the test suite from real failures instead of attempting to enumerate every possible conversation up front.

Verdict

FAQ bots need proof that their defined paths work; AI support agents need that proof plus evidence that answers are grounded and customer data and actions stay within bounds. Pair repeatable tests with exploratory conversations, record enough detail to reproduce failures, and use each defect to improve the next release check.

FAQ

How can a team tell whether its chatbot test set is large enough?

Coverage is more useful than a raw case count. Check that the set includes common intents, varied wording, missing context, unsupported questions, and each high-risk action or data boundary. If a new customer failure cannot be reproduced by an existing case, add a regression case for that exact behavior and its expected safe outcome.

What should happen when a chatbot reaches its conversation limit?

The expected behavior depends on the product’s stated policy and should be tested at the boundary. Zealoop’s pricing page says that at its conversation limit, the agent stops answering, sends visitors a holding reply, adds the conversation to the inbox, and emails the account. Test those outcomes together rather than checking only the visitor-facing message.

When should a team move from a scripted FAQ bot to an AI support agent?

Consider the change when customer questions regularly vary beyond the fixed branches, depend on documentation across several sources, or require a controlled customer-data lookup. Before switching, identify which tasks need source-grounded answers, which require authorization, and where a human must take over; then test those behaviors in the intended workflow.

Sources

• www.zealoop.com
• www.zealoop.com/pricing
• www.zealoop.com/docs/knowledge