Chatbot Testing: Zealoop vs Assistloop, Peopleloop & Loopreply

A support agent can say “I’ve passed this to the team” while the escalation inbox stays empty; checking the sentence alone would miss the failure. Zealoop’s Free plan includes 50 conversations per month, which gives a small SaaS team a defined allowance in which to test real documentation answers and escalation behavior.

Chatbot testing checks whether an AI support agent gives evidence-backed answers and produces the correct system result—not just whether it responds. Scripted assertions are strongest for fixed conditions such as authorization and handoff events; model-assisted evaluation helps score varied wording; human review handles ambiguity. A release passes only when its answer and observable system outcome meet the test case.

The comparison is not a feature-by-feature verdict on Assistloop, Peopleloop, or Loopreply: the available facts do not establish their plan details or the exact behavior of a given deployment. Instead, it shows how a small SaaS team can test Zealoop’s documented behaviors and apply the same outcome-based tests to competing systems. The important distinction is evidence: a fluent response is not proof of a correct answer, authorized lookup, completed action, or successful handoff.

What chatbot testing means for an AI support agent

For a SaaS support agent, chatbot testing validates the full customer interaction against approved information and permitted system behavior. A widget that opens and returns text has passed a basic connectivity check, not a support-quality test. A meaningful test asks whether the answer is supported, whether account data was accessed under the right identity, and whether any claimed action or escalation actually occurred.

A scripted flow and a generative support agent need different assertions. A fixed menu can be tested by selecting an option and checking the next state. An AI agent may express one correct answer in several ways, so an exact sentence comparison can fail valid responses or pass an incomplete answer that happens to include expected words. The test should judge the required facts and outcomes, not reward fluency by itself.

A practical test case separates three layers:

• Conversation: the user’s messages, the agent’s response, and whether it explained uncertainty or a limit.
• Evidence and authorization: the documentation section or permitted customer record that supports the answer.
• System result: a trace, lookup result, action result, or escalation record that confirms what happened outside the response text.

For example, a customer asks whether an order has shipped. A passing result requires an authorized lookup, a response consistent with the returned order status, and no disclosure of another customer’s data. If the lookup fails, the agent must not invent a status. If it says a human has been notified, a handoff record should exist.

That wider test surface is one difference between an FAQ bot and an agent that can use account context. Teams choosing where their tests should begin can compare FAQ chatbots and AI support agents. The core question is not which label a vendor uses; it is what the deployed agent can read, change, and promise to the customer.

What to test: a practical chatbot QA checklist

A small team can cover the main failure modes without building a huge test suite. Each test should name the customer scenario, the acceptable result, and the evidence that proves it.

NLU checks matter even when the agent does not use a fixed intent list. Customers omit product terminology, combine issues, and use short descriptions such as “it’s gone again.” Include paraphrases, misspellings, and ambiguous requests. A useful agent may need to ask for clarification; the test should specify which missing detail prevents a safe answer or action.

Separate performance testing from answer quality. Fast but unsupported responses are not a successful release, and a grounded answer that regularly times out is not a reliable customer path. Run an agreed set of conversations under normal conditions and a load profile chosen for the product. Record latency and technical failures without inventing a universal pass rate.

Security cases should cover prompt injection, attempts to expose hidden instructions, and cross-tenant data requests. For a support agent, it is also useful to test that harmful requests receive a refusal or safe redirection rather than assistance. Keep the expected behavior tied to the product’s support role: the agent should not provide unsafe instructions simply because a customer asked confidently. The security risks of embedded support agents are a useful lens for deciding which access and boundary cases belong in the suite.

How to test documentation-grounded answers

Build answer tests from actual support questions and the current document that establishes each answer. Record the expected facts, relevant conditions, and the source section. Do not keep a reference answer that silently outlives the policy it was copied from. When a source changes, review the cases that depend on it.

A worked example can use a deliberately small test fixture. Suppose a fictional SaaS help article named “Creating an API key” says: “Open Settings, select Developer, then choose Create API key.” The test checks these directions and their citation, not an assumed feature of every SaaS product.

The sample inputs, article, and expected paths above are test-fixture content, not claims about Zealoop or a particular SaaS product. A real team should substitute its own approved documents. Zealoop’s product description says its answers include citations linking to the documentation section used, so a test can inspect the citation as a distinct piece of evidence rather than relying on the wording alone.

Score a generated answer across separate dimensions:

1. Factual accuracy: Are the material claims correct according to the source?
2. Source support: Does the cited section support the claim, or is it merely about the same topic?
3. Completeness and scope: Does the response preserve relevant conditions without adding a promise the document does not make?
4. Uncertainty: When the documentation is missing or ambiguous, does the agent state its limit or abstain instead of guessing?

Exact-string matching is suitable for fixed identifiers or required text, but it is a poor general test for generated answers. “Open Settings, then Developer” and “Go to Developer under Settings” can both express the same supported path. On the other hand, an answer can repeat several words from the article while omitting a condition that changes the customer’s next step. Test meaning and evidence separately.

Include documents with missing, stale, or conflicting answers. For a question with no approved launch date, the expected answer should be a clear limitation rather than a plausible guess. If two current documents disagree, decide whether the agent should ask a clarifying question, explain the conflict, or escalate. Zealoop describes answer validation for grounding; when validation fails, its agent abstains. A regression case can therefore check for the expected abstention rather than treating a missing answer as an automatic malfunction.

How to test customer-data lookups, actions, and escalation

Account-specific behavior needs authorization checks in addition to answer scoring. Zealoop requires a signed identity from the customer’s server for order-history lookups and actions. That is a concrete test boundary: valid identity should permit the configured operation, while missing or invalid identity must not expose account data or authorize an action.

A useful controlled test setup has two tenants, A and B, with distinguishable records. Provide a signed identity for A, ask for A’s order status, and compare the answer to A’s seeded record. Then try to request B’s record while using A’s identity. Pass the first case only if the correct record is returned; pass the second only if B’s data is not disclosed. Repeat the negative case with missing and invalid identity. Keep customer-facing responses and system-side records separate in the test evidence.

Actions need checks for permission, arguments, and result. Suppose a support action updates a subscription setting. The test should verify that the customer is eligible, the action receives the intended account and value, and the external result matches the response. A timeout, denied operation, or unclear result must not produce a success claim. Test a retry as well: if the first result is uncertain, a second customer message should not silently duplicate a change.

Escalation has a similar two-part requirement. The conversation should explain why a human is needed, and a separate observable event should confirm that the handoff reached the expected place. Zealoop lists an escalation inbox and human replies among its Free-plan capabilities. A test can submit a case expected to escalate, verify a handoff appears in that inbox, and check that a human reply can be sent. A sentence saying “the team has been notified” is not proof by itself.

Use explicit negative and recovery cases:

• Missing or invalid signed identity: no account lookup or action is authorized.
• Empty lookup result: the agent reports that the record was not found rather than substituting another record.
• Action timeout: the agent does not claim completion while the result is unknown.
• Permission denial: the agent explains the limitation and follows the defined recovery route.
• Handoff failure: the agent does not claim successful escalation if the handoff event was not created.

A handoff failure example from Adam Innes’s regression-testing discussion makes the distinction concrete: a reassuring message can pass a text check while no one receives the case. The Berkeley technical report on conversation regression testing also treats the interaction as a test unit rather than reducing evaluation to isolated response strings. For a comparison of the different test surfaces, see documentation bots versus guarded support agents.

A repeatable chatbot testing workflow

A manageable release process combines a stable high-risk suite with tests focused on the change. The support regression playbook’s core lesson is that a release should be judged by both conversation behavior and the corresponding system result. A practical sequence looks like this:

1. Define acceptance criteria. Write the facts the answer must contain, the evidence that supports them, and the behavior that must not occur. “Answer the billing question correctly” is too vague; “cite the current billing article, state its renewal condition, and do not claim a charge was reversed” is testable.
2. Create representative and adversarial conversations. Include ordinary phrasing, paraphrases, multi-turn follow-ups, missing documentation, prompt injection, and action failure. Include both happy paths and cases that must be refused, clarified, or abstained from.
3. Select an assertion for each requirement. Use deterministic checks for identity, permission, required fields, and system events. Use semantic evaluation for varied language. Use human review for ambiguous or high-consequence judgments.
4. Run stable and change-focused cases. The stable suite covers important behavior after every release. The focused set covers the documentation, prompt, model configuration, widget, identity handling, or action route that changed.
5. Inspect failures individually. A critical data leak should not disappear inside an average score. Review the answer, source, identity context, and system event to distinguish a real regression from a faulty test.
6. Review a sample of passes. Model-assisted judges can be wrong. Human reviewers should inspect failures and a sample of accepted answers, then adjust the rubric if the evaluator repeatedly misses unsupported claims.
7. Rerun and preserve the regression. After a fix, rerun the failing case and keep it in the suite if it protects important behavior. Record the tested configuration so later runs can be compared meaningfully.

Change scope should determine test scope. A documentation edit should rerun cases tied to the changed facts. A widget update should include browser end-to-end (E2E) checks of the customer path. A change to signed identity, action permissions, or escalation routing deserves the relevant integration cases because the consequence of failure is greater than a wording variation.

For a test that reports a successful handoff, the assertion should inspect the handoff record or inbox state. For a test that reports an order status, it should inspect the lookup result and the identity used. This approach also reflects the practical point in the regression-testing playbook associated with Adam Innes: response text alone cannot establish that an external system completed the promised work.

How to evaluate answers when there is no single expected string

Three evaluation approaches work best together, with each assigned a limited job.

Deterministic assertions check stable facts and events: a citation is present, the expected identity was supplied, a permission was denied, a status field has a defined value, or an escalation record exists. They are reliable for discrete conditions. They are less suited to deciding whether a free-form explanation includes every relevant qualification.

Rubric-based or model-assisted evaluation compares a response with criteria such as evidence support, completeness, uncertainty, and safety. Promptfoo is one named option in the AI-evaluation toolkit; teams can also apply a rubric in their own runner. The evaluator should explain which claims are supported and which are not, rather than awarding points because the answer sounds polished. A model judgment is a signal to validate, not proof that a database action or handoff occurred.

Human review addresses context and impact that automated scoring may miss. A reviewer can decide whether a technically accurate response leaves a misleading impression or whether a particular policy exception changes the outcome. Human review is most valuable for high-risk cases, evaluator disagreements, and periodic samples used to calibrate the scoring rubric.

Score each answer on separate axes:

• Evidence: Are its material claims supported by the approved document or authorized record?
• Intent satisfaction: Did it answer the customer’s question or provide a clear next step?
• Safety and boundaries: Did it avoid unauthorized disclosure, unsafe assistance, and unsupported certainty?
• Action outcome: Did the requested operation succeed, fail, or escalate as expected?

Apply thresholds according to consequence. A false claim that an account action succeeded can be an automatic failure even if the rest of the response is useful. A harmless wording difference can pass if the facts, source, and next step are sound. Calibrate any model judge using known positive and negative examples, then spot-check accepted answers. A high average score cannot establish that a specific critical case is safe.

For a small SaaS support agent, bias testing should be tied to consistent treatment of customers under the same policy. Use paired inputs that differ in irrelevant identity cues or viewpoint while holding the support issue constant; compare whether the agent applies the same product rules. Do not make the test a vague demand for “neutrality.” Define what equal treatment means for that support scenario and review material differences. Medical advice is generally outside product-support scope; if a user asks for it, test that the agent does not invent clinical guidance and follows the product’s defined limitation or escalation path.

Chatbot testing tools compared

No single chatbot testing tool proves every layer. Scripted conversation tools, model-evaluation approaches, browser automation, and human review observe different things. The named options below are useful categories to evaluate; the right choice depends on whether a team can connect the test to its actual widget, source evidence, account data, and system events.

Botium, Promptfoo, Playwright, Google Dialogflow testing, TestMyBot, and the GAP AI Chatbot Testing Framework should not be treated as interchangeable products. A scripted conversation runner is not a browser test; a browser test is not a grounding evaluator; and a model judge is not a record of a completed action. The useful comparison is whether each approach can observe the evidence needed for a given test case.

For a team using Dialogflow, its testing guidance offers an example of keeping intent and fulfillment checks distinct. Botium is a named option for conversational test automation, while Playwright is relevant to browser-based customer-path checks. Promptfoo can be considered for model-oriented evaluation and adversarial inputs. TestMyBot and GAP are also candidates for teams evaluating framework-led approaches. Their exact fit depends on the interfaces, maintenance needs, and evidence available in the team’s environment; this article does not make claims about current pricing or feature parity among them.

A small SaaS team can start with a narrow stack: deterministic assertions for required facts and events, a few browser E2E cases for the real widget, model-assisted review for variable answers, and human review for high-risk failures. That division keeps the most important proof close to the behavior being tested. Teams comparing platform costs can also consult AI customer-support chatbot pricing: FAQ bots vs secure agents, while separately checking the pricing of any test tooling they select.

Metrics and test cases to track over time

Track measures that correspond to customer-facing failures rather than counting replies alone. A useful dashboard distinguishes answer defects from broken system behavior.

• Answer accuracy and grounding: the share of reviewed answers whose material claims match approved documentation or authorized account data. Keep unsupported claims and incorrect citations visible as separate failure types.
• Task completion: whether a lookup or action reached the expected system state, not merely whether the agent said it did.
• Abstention and fallback quality: whether missing evidence leads to a clear limit and useful next step rather than a guess.
• Escalation quality: whether required handoffs were created and carried enough context for a human to respond.
• Response time and technical failures: latency, timeouts, and endpoint errors alongside answer results.
• Security findings: unauthorized access, prompt-injection failures, unsafe assistance, and unintended actions, each tracked by severity.

Do not choose a universal pass-rate target without considering the consequence of failure. A cross-tenant disclosure requires immediate attention even if many ordinary answers were correct. A small change in phrasing may be acceptable when the evidence and outcome remain sound. Teams should set their own release thresholds against product requirements and customer commitments.

For each run, preserve the test ID, input, starting account state, expected evidence, answer, system event, and reviewer decision. Compare results across releases, but do not update an expected answer automatically just because a new model or configuration behaves differently. If an approved document changed, update the expectation with the source change; if behavior drifted without a corresponding product change, investigate it as a regression.

Zealoop provides turn-level traces, which can help teams inspect individual turns when diagnosing an answer or behavior. Its Free plan also gives a concrete small-team starting envelope: $0, 50 conversations per month, and 3 knowledge sources. If a plan reaches its conversation limit, the agent stops answering until the next month or an upgrade, with no overage bill. Include that boundary in release and operational testing: test behavior below the limit and verify what customers experience at the limit rather than assuming a conversation continues normally.

Which should you choose

Choose a test approach based on the agent’s permissions and failure consequences, not on the length of a vendor feature list.

• For a documentation-only FAQ bot: start with source-linked questions, citation checks, missing-answer cases, and a small set of paraphrases. Add browser checks if the embedded widget is a critical customer path.
• For a documentation-grounded agent such as Zealoop: combine citation and grounding checks with explicit abstention cases. Use turn-level traces to investigate individual failures, then retain the case that exposed the defect as a regression test.
• For an agent that reads customer data: make signed identity and tenant isolation hard pass/fail requirements. In Zealoop’s case, order-history lookups and actions require server-signed identity, so test valid, missing, and invalid identity conditions rather than only the successful lookup.
• For an agent that can take actions: test permissions, arguments, result states, retries, and failure messages. Require system-side evidence before passing a customer-facing success claim.
• For teams with limited QA time: automate a small number of high-risk deterministic cases first, then use a rubric and human review for variable answers. Choose a tool only when it can observe the evidence those cases require.
• For a comparison with Assistloop, Peopleloop, or Loopreply: run the same customer scenarios against the actual configured products. Record citations, abstention, identity requirements, action results, and handoff events separately; do not infer those details from the shared support-automation category.

For a small SaaS team using Zealoop, the practical starting point is a stable regression suite with three layers: document-grounded answer cases, signed-identity and action cases, and escalation verification. The Free plan’s 50 monthly conversations and 3 knowledge sources make it possible to try a documentation-answering setup without a time-limited demo; the hard stop at the plan limit should also be part of the test plan. Add model-assisted scoring only where fixed assertions cannot judge meaning, and retain human review for consequential disagreements.

Verdict

Zealoop is best tested as an agent with both evidence and system boundaries: check citations and abstention for documentation answers, signed identity for order lookups and actions, and an actual inbox event for escalation. For the alternatives, use the same scenarios and compare the observed outcomes in each deployment. Scripted checks prove fixed conditions, model-assisted evaluation scores language, and human review resolves consequential ambiguity.

FAQ

What is chatbot testing, and why does a SaaS team need it?

Chatbot testing checks whether conversations produce correct, safe, and useful outcomes, including system effects outside the reply. It matters because “your order was updated” can be false even when the text sounds convincing. For a support agent, tests should verify source evidence, identity, action results, and handoffs as well as the customer-visible answer.

How should a team test answers when the wording can vary?

Record required facts, source evidence, and conditions instead of one exact paragraph. For example, a help-center answer may state the same Settings → Developer path with different wording; both can pass if the citation supports it. Use deterministic checks for stable facts, rubric-based evaluation for meaning, and human review for disputed or consequential cases.

What happens when Zealoop reaches its Free-plan conversation limit?

The agent stops answering until the next month or an upgrade, and there is no overage bill. A team should include a limit-boundary case in its operational testing so it understands the customer experience when the allowance is reached. The Free plan is listed at $0 with 50 conversations per month and 3 knowledge sources.

How can a team prove an escalation worked?

Check for a system-side handoff record in the expected destination, not just a response saying a human was notified. A complete test can confirm that the thread appears in the escalation inbox and that the conversation includes enough context for the human to respond. If the record is absent, the test fails even if the wording sounds reassuring.

Sources

• www.zealoop.com
• www.zealoop.com/pricing
• www.zealoop.com/compare/faq-chatbot-vs-ai-support-agent
• www.zealoop.com/compare/ai-chatbot-security-risks-support-agents
• www.zealoop.com/compare/ai-customer-support-chatbot-docs-vs-agents
• www.zealoop.com/compare/ai-customer-support-chatbot-pricing
• assistloop.ai/blog/chatbot-support-regression-testing
• botium-docs.readthedocs.io/en/latest/03_testing/01_testing_conversational_ai.html
• developers.google.com/assistant/conversational/testing-best-practices