Human Review vs Automated Tests: How to Measure Chatbot Accuracy

A support agent can return the correct account field and still tell the customer the wrong thing about what happens next. Testing only the lookup misses the answer; reviewing only the wording misses whether the lookup worked. Knowing how to measure chatbot accuracy means comparing those two forms of evidence against a defined expected outcome.

Chatbot accuracy is the percentage of eligible test units that pass criteria defined for their task. Human review is strongest at judging context, meaning, and safe handling; automated tests are strongest at checking repeatable conditions such as exact values, required confirmations, and action results. A useful evaluation combines them without treating either as proof of overall support quality.

How to measure chatbot accuracy—and why one score is not enough

Accuracy means correctness against an explicit expected outcome. The expected outcome depends on the task: a documentation answer should be accurate and supported; an intent classifier should select the right category; a customer-data lookup should return the right customer’s value; and an action should respect its permissions and preconditions. A fluent reply can fail any of these tests.

For a support agent, keep these dimensions distinct:

• Factual correctness: Does the response match the customer’s question and situation?
• Grounding: Can its material claims be traced to approved documentation or verified customer data?
• Intent understanding: Did the system identify what the customer is actually trying to do?
• Task completion: Did the lookup or requested action produce the expected result?
• Safe behavior: Did the agent respect identity, permission, confirmation, and escalation requirements?

These dimensions can disagree. An answer may state the right general policy but fail to check an account-specific condition. A tool call may succeed against the wrong account. A handoff may leave the question unanswered by the bot but still be the correct safe decision.

The evaluation method should match the evidence available. A reviewer can judge whether a nuanced answer applies to the thread; a script can verify whether a specific field equals an expected value. Neither alone answers whether the customer’s issue was resolved. Report task-level measures first, and use an overall figure only with its scope and component scores visible.

For example, a 40-case test set might contain 20 documentation answers, 10 account lookups, 6 guarded actions, and 4 escalation cases. A single percentage can hide that every failure occurred in account lookups. Reporting the result by workflow points the team toward the defect instead of suggesting that all capabilities perform alike.

Choose metrics that match the chatbot’s job

A metric is meaningful only when its unit and denominator fit the job. Decide whether the unit is a message, a tool action, or a complete conversation. Message-level results expose individual turn errors; conversation-level results capture whether a multi-turn issue was handled safely from start to finish.

Classification and intent detection

For labeled intent examples, classification accuracy is:

Correctly classified examples ÷ all labeled examples × 100

If a classifier assigns the right intent to 45 of 50 messages, its accuracy on that set is 90%. That number can conceal a weak result on a less common but high-impact intent. If the 50 messages include only five access-recovery cases and the system misses three, report that class separately.

Use precision and recall when a particular category has different costs for false positives and false negatives. Precision asks how many messages labeled “account access” really belong there. Recall asks how many actual account-access requests the system identified. Missing an access request may leave a user in an irrelevant automated flow; over-labeling it may create unnecessary escalation.

Generated answers, lookups, and actions

For open-ended answers, score human-rated correctness and source support separately. A correct answer may have no approved evidence behind a material claim; a well-supported answer may omit an important condition. For lookups, compare the returned customer field with the expected value. For actions, check the requested operation, target identity, permission, confirmation, and resulting state—not merely whether an API call returned successfully.

Containment, resolution, and latency are useful companion measures, not accuracy measures. Containment describes whether a conversation remained automated under a team’s definition. Resolution asks whether the customer’s issue was addressed, including through a human handoff. Latency measures time. A fast, contained response may still be wrong, while a correct escalation can be the best outcome even though the chatbot did not complete the answer itself.

A support team can make the denominator explicit for each metric: answer correctness among eligible answers, lookup success among eligible lookup requests, and safe-action success among eligible action requests. Combining those rates into one score requires a stated weighting rule; otherwise separate results are easier to interpret and harder to distort.

Human review vs automated tests: what each method can detect

Human review and automated tests are complementary, not interchangeable. A human can interpret meaning and context but has limited throughput and may disagree with another reviewer. Automation can repeat precise checks consistently but only checks what its rules, reference data, or judge prompt can recognize.

Human review: context, meaning, and risk

A reviewer can see that a customer’s latest question depends on an earlier message, compare the reply with approved documentation, and decide whether the agent should have asked for clarification. That makes human review valuable when answers have multiple acceptable phrasings or when a small mistake could mislead a customer.

Its limitations are reviewer time and inconsistent interpretation. Two people may disagree about whether a missing detail is material or whether a handoff was necessary. A written rubric and a shared set of examples reduce that inconsistency; they do not eliminate the need to inspect disagreements.

Deterministic tests: explicit conditions

A deterministic test checks a condition with a defined expected result. Examples include whether an intent label matches its expected class, whether an account lookup returns the expected field, whether a guarded action uses the correct identifier, and whether a required confirmation occurs first.

Such tests are repeatable and useful in release checks. They are less suitable for judging a natural-language paraphrase, an incomplete but safe explanation, or a decision that depends on several conversation turns. A test that searches for a required phrase might pass a reply that includes the phrase alongside a contradiction. Rules should verify objective conditions, not stand in for review of every answer.

Embeddings and LLM judges: useful signals, not ground truth

Embedding similarity estimates whether two texts are semantically close. It can help group similar answers or flag a response that differs substantially from a reference. It can also miss a factual reversal: “the payment succeeded” and “the payment did not succeed” share most of their words. Similarity is therefore a screening signal, not evidence that a customer-specific claim is true.

An LLM-as-a-judge can apply written criteria to varied answers, which is useful when exact string matching is inappropriate. But a judge can reward fluent writing while missing a factual error, and its verdict may change with its prompt or model. Give it the relevant conversation and approved evidence, then compare its labels with human decisions before relying on it at scale.

DeepEval is one example of an evaluation framework. The choice of framework does not determine whether an evaluation is valid: the test cases, expected outcomes, rubric, and review of disagreements do. Teams can also begin with a spreadsheet and small scripts, then adopt a framework when maintaining cases or repeatable runs becomes burdensome.

Design a support evaluation set around workflows

A useful dataset represents the support work the agent is expected to do. It should not consist only of easy frequently asked questions. Build a controlled set for repeatable release tests and a separate sample of live conversations for discovering how customers phrase requests in practice.

Start with these four workflow groups:

1. Documentation answers: product behavior, setup instructions, policies, and known limitations.
2. Customer-data lookups: questions whose answers depend on a specific account or subscription.
3. Guarded actions: changes that require the right identity, permission, confirmation, or other precondition.
4. Escalation: ambiguous, unsupported, sensitive, or otherwise human-dependent requests.

Include common and edge-case requests in each relevant group. Add multi-turn cases where a customer clarifies, changes a detail, repeats a question, or contradicts earlier information. Include out-of-scope questions and cases where the correct outcome is to ask for more information rather than guess.

For each example, preserve the context a fair reviewer needs: the relevant customer turns, the agent’s response, the approved source or expected account result, and the expected action or handoff. Record an example ID, workflow, intent, expected outcome, observed response, reviewer label, and failure category. Remove unnecessary personal information from review records.

A small team can start with 50 cases, deliberately spread across its four workflows rather than sampling only the most common FAQ. That is a starting test set, not a statistically guaranteed estimate of production accuracy. If a team reviews 50 cases at an assumed three minutes each, the first pass takes 150 minutes—two and a half reviewer-hours. The three-minute figure is a planning assumption; actual time varies with conversation complexity.

Keep ambiguous cases visible. Mark them “needs clarification” or “expected outcome unclear,” then decide in advance whether they are excluded from an accuracy denominator or scored as a required clarification. State the count and treatment in the report. Quietly dropping hard cases after seeing the answer can make a result look better without making the chatbot better.

Score answers consistently and measure reviewer agreement

A rubric should define what a pass means before reviewers see the results. These criteria cover different failure modes and can be scored independently:

• Correctness: The answer matches the customer’s actual request and relevant facts.
• Source support: Material claims are supported by approved documentation or verified data.
• Completeness: The response includes conditions or next steps needed to act correctly.
• Uncertainty: Missing information triggers a question or safe handoff rather than a guess.
• Policy compliance: Identity, permission, and confirmation rules are respected.
• Action or handoff: The expected operation succeeds, or the appropriate human receives the case and useful context.

For example, a documentation answer that gets the main step right but leaves out a nonessential explanation might pass correctness and fail completeness. A lookup that returns another customer’s value fails correctness and safe behavior even if the sentence is clear. Keep the component labels; avoid letting a broad “good answer” rating conceal the kind of failure.

Before scaling review, have two reviewers independently label the same pilot cases. Suppose they both label 24 cases pass, both label 6 fail, and disagree on 10 cases: one reviewer marks 4 pass and 6 fail where the other does the opposite. They agree on 30 of 40 cases, so observed agreement is 30 ÷ 40 = 75%.

Cohen’s kappa adjusts that figure for agreement expected from the reviewers’ pass/fail rates. In this example, the first reviewer marks 28 of 40 pass (70%) and the second marks 30 of 40 pass (75%). Expected chance agreement is (0.70 × 0.75) + (0.30 × 0.25) = 0.60. Therefore, κ = (0.75 − 0.60) ÷ (1 − 0.60) = 0.375. The gap between 75% raw agreement and κ of 0.375 shows why raw agreement alone can overstate consistency.

As a practical internal gate, a team can aim for κ ≥ 0.70 on the pilot labels before using those labels to calibrate automated scoring. It is an operating threshold, not a universal proof of rubric quality. Below it, inspect disagreements, sharpen examples and definitions, and score a fresh subset. Even above it, examine high-impact disagreements individually.

Calculate rates and inspect errors by category

For each reported rate, state the unit, numerator, denominator, and exclusions. A simple answer-level pass rate is:

Accuracy = passing eligible answers ÷ all eligible answers reviewed × 100

Suppose a 50-case evaluation has 20 documentation answers, 12 lookups, 10 guarded actions, and 8 escalation decisions. The first review finds 16 documentation answers passing, 9 lookups returning the expected value, 7 actions meeting all preconditions and producing the expected result, and 7 appropriate escalation decisions. That is 39 passes out of 50, or 78% overall.

The overall result is not enough to guide a fix. By workflow, the rates are 16/20 = 80% for documentation, 9/12 = 75% for lookups, 7/10 = 70% for guarded actions, and 7/8 = 87.5% for escalation decisions. The action result is the weakest category, so the team should inspect target identity, confirmation handling, and resulting state before changing documentation answers.

If three of the 50 cases had corrupted transcripts, and the team’s prewritten rule excludes corrupted records, report 39 passes out of 47 eligible cases (83%) and disclose the three excluded cases. Do not exclude an answer just because reviewers disagree or the expected outcome is awkward. Mark it separately, resolve whether the case is valid, and report how it was treated.

Track counts alongside rates. With a small set, one result can move the percentage sharply: one case changes a 50-case score by two percentage points. Include the test-set version, sample period, workflow counts, failure categories, and representative examples. A rate without those details is difficult to compare across releases or diagnose.

Connect correctness to resolution, containment, and latency

Accuracy measures whether the evaluated response or action met its criteria. Resolution measures whether the customer’s issue was addressed; containment indicates whether the conversation stayed automated; escalation describes whether it reached a human appropriately; latency measures elapsed time. These outcomes should be read together, not folded into an unexplained “quality” score.

If containment rises while correctness or resolution falls, investigate what changed in the conversations that no longer escalated. Review a sample of those cases for unsupported claims, unresolved questions, repeat contacts, and missed handoffs. More automation is not an improvement if customers remain with the bot but leave with an incorrect answer.

A rise in escalation can also be healthy. If the agent now hands off account disputes it previously guessed at, containment may decrease while safe handling and eventual resolution improve. Conversely, faster responses do not validate an answer, and positive satisfaction scores do not prove that a customer-specific lookup was correct. Use those signals to identify cases to inspect, then review the thread and evidence.

For example, a team that sees faster responses and higher containment after a release should compare the same workflow-level test set and sample newly contained live threads. If action accuracy falls from 7/10 to 5/10 in a matched test, the speed improvement does not offset the two additional action failures. The next step is to locate the action precondition or identity check that failed.

Keep human and automated evaluation in a release cycle

A compact cycle avoids repeating the entire evaluation from scratch. Keep a versioned baseline set for regression tests, then add representative cases when policies, data fields, documentation, or actions change. A refund-policy edit calls for affected policy cases; a new account action calls for tests of identity, permissions, confirmation, and final state.

A workable release routine is:

1. Run deterministic checks and the fixed baseline against the proposed change.
2. Review failures in the affected workflow, including source evidence and action traces where available.
3. Double-score a small subset when the rubric, agent behavior, or judge changes.
4. Sample live conversations for new phrasing, unclear cases, and unexpected failures.
5. Add valid new cases to the set and retain the dataset and rubric versions with the result.

Zealoop provides a practical example of how product evidence can support that review. Its Free plan is a permanent $0 plan with 50 conversations per month and 3 knowledge sources. A conversation means one thread with one visitor, regardless of its message count. The plan includes cited answers, signed identity (HMAC), an escalation inbox, human replies, and turn-level traces. Those features provide review artifacts: a citation can be checked against its source, a trace can help locate a problematic turn, and the inbox allows a human to inspect escalated threads. They do not themselves establish that an answer is correct.

Zealoop describes its retrieval as hybrid vector and full-text search followed by a cross-encoder reranker. For evaluation, that makes it useful to distinguish the final answer from the evidence shown for it: reviewers can assess whether the cited support is relevant and whether the response accurately represents it. This is a product-specific review opportunity, not a general guarantee about retrieval quality. The distinction between documentation answers and controlled workflows is also covered in this comparison of documentation bots and guarded support agents.

A spreadsheet is enough for a small pilot; an evaluation framework such as DeepEval may help organize repeatable tests as the set grows. The important release artifacts remain the same: fixed inputs, expected outcomes, visible failure categories, and human review of cases where automated checks cannot judge the context.

Which should you choose

For small SaaS support teams, the strongest default is human-labeled review plus deterministic tests, with embeddings or an LLM judge used to prioritize or extend review—not replace it.

• Choose human review first when the team is defining what “correct,” “grounded,” or “appropriate escalation” means for its product.
• Choose deterministic tests for repeatable, objective conditions: exact lookup values, intent labels, permission checks, and required confirmation steps.
• Add embedding similarity to sort or flag related responses, not to approve factual claims.
• Add an LLM judge when a varied set of open-ended answers makes manual review expensive, and calibrate it against human labels before trusting its rates.
• Score multi-turn conversations as well as individual turns when earlier messages can change the correct answer.

A small team with a limited budget can begin with a 50-case stratified set and a human-reviewed pilot, then automate the checks whose expected results are unambiguous. A team evaluating documentation-grounded answers should keep source support separate from factual correctness. A team enabling account changes should prioritize action-result and safe-behavior checks, even if that means more human inspection.

Verdict

Human review is the better method for establishing context-sensitive labels; automated tests are better for repeating explicit checks at release time. A chatbot accuracy program becomes useful when the methods are compared against the same expected outcomes, disagreement is measured, and results are reported by workflow. Use automation to make review more repeatable—not to turn fluent answers into assumed truth.

FAQ

How can a small SaaS team measure chatbot accuracy for free?

A spreadsheet, a written rubric, and a small sample are enough to begin. Record the expected outcome, response, evidence, label, and failure category; use short scripts only for exact checks. Zealoop’s Free plan is $0 forever and includes 50 conversations per month and 3 knowledge sources, providing a bounded set of real support threads for review rather than a paid evaluation requirement.

How many conversations should a team review each month?

There is no single sample size that guarantees a reliable estimate across all support workloads. A team can start with 50 cases spread across documentation answers, account lookups, actions, and escalation, then adjust based on volume and risk. Keep high-impact workflows represented even when they are less common, and report category counts so the overall rate does not hide a weak area.

When should a team switch from manual scoring to an LLM judge?

Add a judge when the human rubric is stable and review volume is becoming difficult to sustain. First compare its pass/fail results with human labels on the same cases, including borderline and high-impact examples. If disagreement clusters around source support or multi-turn context, improve the judge inputs or keep those categories under human review rather than expanding automation blindly.

What should happen when reviewers disagree about an answer?

Keep the disagreement as evaluation data instead of silently forcing a label. Compare the customer context, approved evidence, and rubric language; then decide whether the example needs a clearer expected outcome, a clarification label, or a rubric revision. Re-score a fresh subset after changes so improved agreement is not just memorization of the cases reviewers already discussed.

Sources

• www.zealoop.com/
• www.zealoop.com/pricing
• www.zealoop.com/compare/ai-customer-support-chatbot-docs-vs-agents
• deepeval.com/docs/getting-started-chatbots