Chatbot A/B Testing — Two Versions, One Outcome, Decided in Advance

Quick answer: In a chatbot A/B test, version A (usually what you have now) and version B (one change) run side by side. Each new conversation is assigned to one of them at random. After enough conversations you compare one pre-chosen outcome, and only then decide whether B really did better or the gap is noise. The common mistakes are testing several changes at once, stopping when the numbers look good, and judging by a metric that was not chosen beforehand.

What you can test

Almost any customer-facing choice can be a variant, provided you change one thing at a time.

Prompt changes need extra care: the system prompt changes every answer, so also read sample conversations from both arms instead of trusting one number.

Choosing the metric

A test needs one primary metric named before it starts, plus a few guardrail metrics that must not get worse. Typical primary metrics are the conversion rate of a step, the flow completion rate or the containment rate. Typical guardrails are CSAT, handoff rate and complaints. A variant that raises sign-ups while pushing satisfaction down has not won.

Sample size and significance

Two terms matter. Sample size is how many conversations each version needs before the comparison can be trusted. Statistical significance is how unlikely a gap this large would be if the two versions were really identical.

For a rate such as lead capture, the standard calculation uses the baseline rate, the smallest improvement you care about, a significance level (commonly 5%) and power (commonly 80%). Our recomputation with those conventional settings: to detect a rise from 30% to 36%, you need about 961 conversations per version, so roughly 1,922 in total. Smaller expected improvements need far more. The settings are conventions, not laws, and a calculator or a statistician should confirm them for your case.

Once the planned sample is in, a two-proportion z-test compares the two rates. For a two-by-two table this gives the same verdict as a chi-squared test, since the chi-squared statistic equals the square of the z value.

Why you must not stop when it looks good

Checking the results every day and stopping the moment they look significant raises the false-positive rate. Evan Miller's article puts it plainly: "Repeated significance testing always increases the rate of false positives." His remedy is to "decide on a sample size in advance and wait until the experiment is over." Chatbot traffic is often small, so the temptation to peek is strong, and so is the cost of acting on a fluke.

How tools implement it

Intercom's help articles, read on 25 October 2026, show one real pattern.

• Message A/B tests. A test compares two versions, A and B. Both are sent at the same time and assignment is random, so the split is not guaranteed to be 50/50; small audiences and a version that went live earlier can skew it. The article says it does not state a minimum sample size or test length. You can pause the losing version so the winner reaches everyone.
• Split testing in Series. A split block can have up to five paths, you set the percentage entering each (which must total 100%), each test has its own goal, and you can add a control group that receives no messages. Results appear on the split block, and declaring a winner pauses the others. The feature needs a paid plan.

The takeaway is not that one tool is better. It is that the tool assigns and reports, but you still decide the metric, the sample size and the stopping point. (Intercom is shown only as a documented example; our hands-on Intercom review covers the product itself.)

Ways a chatbot test misleads

These are our reasoning from how experiments work, not measured rates.

• Several changes at once. If B has a new greeting and a new button order, you cannot tell which mattered.
• Unequal exposure. One version live earlier, or a channel that reaches only one arm, breaks the comparison.
• Repeat visitors. The same person can see both versions if assignment is per message rather than per person.
• Small samples. A 10-point swing on 40 conversations is easily chance.
• Novelty. A new message can win for a week simply because it is new.
• Time effects. Comparing this week's B with last week's A is not an A/B test; run both at the same time.

Worked example (invented numbers)

A bot asks for an email at the end of a pricing chat. Today 30% of conversations that reach that step leave one. B moves the ask earlier.

• Plan: detect 30% to 36%, about 961 per version, 1,922 total. At 100 qualifying chats a day that is 1,922 ÷ 100 = 19.2, so 20 days.
• Result A: 300 of 1,000 = 30.0%. Result B: 350 of 1,000 = 35.0%. The z value is about 2.39 and the two-sided p-value about 0.017, so a gap this size would be unusual if the versions were equal.
• Different outcome: if B had 330 of 1,000 (33.0%), z would be about 1.44 and p about 0.15. Three points better looks nice but is not enough evidence at these numbers.

All figures are invented for illustration and the arithmetic was checked.

FAQ

What is A/B testing in a chatbot?

Showing two versions of one element to different customers at the same time and comparing one outcome. Assignment is random so the groups are alike.

What should I A/B test first in a chatbot?

Start with the step that most customers reach and where many drop out, often the welcome message or the first question, because it gives you the most conversations per day.

How many conversations do I need?

It depends on your baseline rate and the smallest gain you care about. To go from 30% to 36% at conventional settings, about 961 per version. Use a sample-size calculator for your own numbers.

Can I stop a test early if one version is clearly ahead?

Not safely. Repeated checking inflates false positives. Fix the sample size first and stop only when it is reached.

Is A/B testing the same as multivariate testing?

No. A/B changes one element. Multivariate tests several at once and needs much more traffic.

Does it work for AI answers, not just fixed messages?

Yes, but answers vary, so also read sample conversations from both arms and watch guardrail metrics.

Related terms

• Chatbot conversion rate — the most common primary metric.
• Chatbot analytics — where the numbers come from.
• Chatbot greeting message — the most tested element.
• Quick reply — button labels are a frequent variant.
• Phased rollout — releasing to a share of users, related but not a controlled comparison.
• Golden dataset — the offline alternative for checking AI answers.

Sources

• Intercom, A/B test different message variations — intercom.com/help/en/articles/5296707-a-b-test-different-message-variations, read 25 October 2026. Source of the concurrent send, random assignment and pause-the-loser behavior.
• Intercom, Test different paths with multivariate split testing in Series — intercom.com/help/en/articles/4730329-test-different-paths-with-multivariate-split-testing-in-series, read 25 October 2026. Source of the five-path limit, percentages, control group and paid-plan note.
• Evan Miller, How Not to Run an A/B Test — evanmiller.org/how-not-to-run-an-ab-test.html, read 25 October 2026. Source of the quoted statements on repeated testing and fixed sample size.
• Chatbotscape evaluation methodology. /methodology (continuously updated).