How to A/B Test Your Chatbot's Messages and Flows

Quick answer: Change one thing, name one metric, work out how many conversations you need before you start, split new conversations randomly and at the same time, run until you reach that number, then compare. If B wins by more than chance would explain and your guardrail metrics held, keep B. If not, keep A and test something else. Stopping early because B looks ahead is the most common way to fool yourself.

Step 1: Write down the question

Start with a sentence, not a setting: "Does asking for the email before the price answer get more people to leave one than asking after?" One question, one change. If you want to try a new greeting and a new button order, that is two tests.

Pick a place with enough traffic. A step that only 20 people reach a month cannot be tested in any reasonable time. Look in your analytics for the step where many people arrive and many drop out, often the first message. The guide to welcome message design lists ideas for variants, and button versus free text covers another common one.

Step 2: Choose one primary metric and a few guardrails

The primary metric decides the winner and is fixed before you start. Make it a rate with a clear denominator: of the conversations that reach this step, what share leave an email, click a button or finish the flow. Our conversion rate entry explains how to define the denominator.

Add guardrails: numbers that must not get worse, such as satisfaction (improve chatbot CSAT), handoff rate and complaints. A variant that adds sign-ups while customers get angrier has not won. Write all of this in one line in your test log.

Step 3: Work out the sample size before you launch

You need four inputs: your current rate, the smallest gain worth acting on, a significance level and power. Common conventions are 5% and 80%. These are conventions, not guarantees, and an online sample-size calculator or a statistician can confirm them.

Our recomputation for a rise from 30% to 36% with those settings is about 961 conversations per version, 1,922 in total. Divide by your daily traffic at that step to get a duration. At 100 qualifying chats a day, 1,922 ÷ 100 = 19.2, so plan 20 days. If the answer is six months, test a bigger change or a busier step.

Smaller gains need many more conversations. Halving the gain you want to detect roughly quadruples the sample, which is why tiny wording tweaks are poor candidates for a small bot.

Step 4: Set up a clean split

How you assign people matters as much as the maths.

• Run A and B at the same time. Comparing this week with last week is not an A/B test.
• Assign randomly, once per person. If someone returns tomorrow, they should see the same version.
• Keep everything else equal. Same channel, same hours, same audience rules.

Look at what your tool does. Intercom's documentation, read on 25 October 2026, says both versions of a message are sent concurrently and assignment is random, so the split may not be exactly 50/50, and small audiences, or a version that went live earlier, can skew it. Its Series split block lets you set a percentage for each of up to five paths, add a control group that gets no messages, and give each split its own goal. It requires a paid plan. Other tools differ, so read yours; our ManyChat review and Intercom review cover those products generally, not their testing features. If your tool has no split feature, you can alternate by a rule that is unrelated to the customer, such as a random flag set when the chat starts, and tag each chat with its version.

Do not edit either version once the test is live. Changing text mid-test mixes two experiments.

Step 5: Run it to the planned end

Look at the dashboard if you must, but do not decide anything until you reach the planned sample. Evan Miller's article on this subject says: "Repeated significance testing always increases the rate of false positives." His advice is to "decide on a sample size in advance and wait until the experiment is over."

During the run, only watch for breakage: a link that fails, a variant that crashes, a flood of complaints. Stop for those and restart cleanly. Also run full weeks where you can, so weekday and weekend visitors both appear in each arm.

Step 6: Read the result

When the sample is complete, take the primary metric for each version and compare. A two-proportion z-test is the standard check; many calculators ask only for the number of visitors and successes per version.

Then work through three questions:

1. Is the gap bigger than chance would explain at your chosen level?
2. Did the guardrail metrics hold?
3. Is the gain large enough to matter in money or time?

Read ten to twenty conversations from each version as well. Numbers show that something changed; transcripts show why, and catch problems the metric misses. Record the outcome in the test log, including the tests that did not win, then pick the next question. Keep the winner as a named, dated version, as in versioning and releasing chatbot prompts.

Check the arithmetic

An invented test. The bot asks for an email at the end of a pricing chat. 30% leave one now. B moves the ask earlier.

• The plan called for about 961 per version, so 1,000 each meets it.
• The z value is about 2.39 and the two-sided p-value about 0.017. A gap this large would be unusual if the versions were equal.
• Contrast: had B left 330 emails (33.0%), z would be about 1.44 and p about 0.15. That is not enough evidence at this sample, however pleasing the three-point gap looks.
• Cost view: 50 extra emails per 1,000 chats. At 100 chats a day that is about 5 extra emails a day, or about 150 in 30 days. If one email in ten becomes a $200 sale, that is 15 × $200 = $3,000, a made-up conversion rate used only to show how to translate the gain.

All numbers are invented and the arithmetic was checked.

Common mistakes

• Changing two things at once. You will not know which one worked.
• Peeking and stopping. See Step 5.
• Choosing the metric afterwards. Searching for any number that moved guarantees a "winner".
• Testing tiny changes on a tiny bot. The sample never arrives.
• Ignoring AI variation. If the variant is a prompt change, answers differ every time, so read samples.
• Forgetting novelty. A fresh message can win for a week. Re-check the winner a month later.

FAQ

How do I A/B test a chatbot?

Pick one change, one primary metric and a sample size. Randomly show A or B to new conversations at the same time, run to the planned number, then compare the rates.

What should I test first, the welcome message or the button labels?

Whichever step more people reach and abandon. That is usually the welcome message, since everyone sees it, which gives you the sample fastest.

How many conversations do I need?

It depends on your current rate and the smallest gain you care about. Going from 30% to 36% needs about 961 per version at conventional settings.

How do I know if the result is statistically significant?

Run a two-proportion z-test, or use a calculator, on the visitors and successes per version once the planned sample is in. A small p-value means a gap that large would be unusual if the versions were equal.

Which chatbot tools support A/B testing?

Some do. Intercom documents message A/B tests and Series split testing. Check your own tool's current documentation, and use a manual random flag if it has no feature.

Can I stop early if one version is far ahead?

Not safely. Checking repeatedly and stopping when it looks good raises false positives. Stop early only for breakage.