Chatbot Regression Testing

How to Change a Prompt, a Knowledge Base or a Flow Without Breaking What Already Worked

Quick answer: A live chatbot changes every week: a help article is edited, a line is added to the system prompt, the vendor swaps the model underneath. Each change can fix the thing you touched and break three things you did not, and nobody notices until the fallback rate climbs or a customer posts the wrong refund policy. Regression testing is the habit that catches it: keep a frozen set of 30 to 50 questions with known-correct outcomes (a golden dataset), run it before the change and again after, and read the difference. The vendors now supply the harness. Read on 7 September 2026, Intercom runs up to 50 questions per group free of charge and returns the sources Fin used; its Simulations play an AI customer against a Procedure and return a pass or fail; Voiceflow has both a scripted test runner with per-check pass or fail and Evaluations that score every transcript against criteria you write; Botpress, Chatbase and Tidio give you a test console and leave the grading to you. This guide is the procedure around those tools: what counts as a change, how to build the set, which harness to use, how to read the diff, how to triage, and a cadence a one-person team can keep.

Step 1. Decide what counts as a change

The failure that makes regression testing necessary is that the edit and the breakage are in different places. For example, a knowledge-base article about shipping is rewritten, and the returns answer changes because the two articles shared a paragraph the retrieval step used to rank. A sentence is added to the guidance to make the bot more concise, and the multi-step password reset loses its third step. So the trigger list is written by what can move the answer, not by what you intended to change.

The rule that follows: content, prompt and model changes get the full set; flow and integration changes get the tagged subset plus the handoff rows; and the calendar gets a full run once a month whether or not anything changed, because the last row of the table is the one nobody schedules.

Step 2. Build the set you will run every time

The golden dataset entry has the anatomy; the short version is that a row is an input, an expected outcome stated as what must be true rather than what must be said, a must-not, tags for topic, language and channel, a source of truth with a date, an owner, and the last result. Build it once from four sources, in this order.

Start with past conversations, because they are what customers ask. Intercom generates "up to 50 questions based on your most recent customer conversations (30-90 days)" and, with AI Topics enabled, by topic; on any other platform, export the last month of transcripts and pull the distinct questions by hand. Add the escalations the bot did not handle, deduplicated by intent. Add the rows no transcript contains: the feature launching next week, the policy that changes on the first of the month, the compliance question. Intercom's article names this set ("compliance-critical scenarios and/or edge cases that don't appear in historical conversation data") and gives it the manual and CSV paths. Finish with a quarter of the set that should not be answered: the refusal, the clarifying question, the handoff, the prompt-injection attempt.

Then write the expected outcome for every row yourself. This is the hour nobody wants to spend and the only part no tool does. Our own reviews use 15 questions per platform for the first-week check; the Tidio review ran 15 against Lyro's Playground and got 11 grounded answers, 3 correct refusals and 1 answer that "conflated two unrelated FAQ items," and calls the result "a first-week snapshot, not a steady-state measurement." Fifteen finds the failure class; thirty to fifty, three to five per topic, is where a regression becomes visible by topic.

Step 3. Pick the harness, and know what it cannot see

Each platform gives you one or more of three things: a console to type into, a batch runner that answers many rows at once, and a judge that returns a verdict. The table is what each gives and, more usefully, what each withholds, read on 7 September 2026.

Two consequences. First, a "batch" on Botpress, Chatbase or Tidio is you, typing 40 rows, which is why Botpress's ↑ key to resend earlier messages is the most useful feature in its row; budget 45 to 60 minutes to run, up to 90 with manual grading, and keep the rows in a sheet you can paste from. Second, the sandboxes are sandboxes on purpose: nothing in the Intercom Batch test or a Simulation touches Shopify, Stripe or your CRM, so the integration rows in your set can only be truly run in Preview mode against a test user, and the Intercom comparison article says exactly that: "If you need to test with real API calls, use Preview and select a specific user with real data in their external system." Mark those rows in the set so nobody counts a sample-payload pass as a live pass.

Step 4. Baseline, change, re-run, diff

The run is the same four moves every time. Voiceflow publishes them as its own best practice for "any meaningful change to your agent": clone Main into a new environment, make the change there, add tests only for new behavior, click Run all tests so that "existing tests catch behaviour you didn't mean to change," iterate until green, then merge back to Main or route some traffic to the new environment first. That is the procedure below with the platform's names on it.

Baseline before the change. Run the full set (or the tagged subset from step 1) on the bot as it is. Record a result per row: pass, fail, or the reason code. If you skip this and only run after, you cannot tell a regression from a failure that was already there; on a set you have run before, the last run's results are the baseline and this step is free.

Make one change. Two changes in one run means a regression with two suspects. Where the platform batches changes for you (a vendor model upgrade lands with a content edit you made the same day), record both and expect to test twice.

Re-run the same set. Same rows, same test-as settings. Intercom's test groups keep the settings ("each test group retains the settings you used during testing—like simulating a specific user—so you can re-run tests with the same configuration anytime"); on a manual console, the same person should run both rounds if you can manage it, because a second grader is a second variable. The Batch test article's FAQ notes the distinction between resetting and re-running: re-running "will re-generate answers based on any content changes," which is the operation you want.

Diff by row, then by tag. Three outcomes per row: unchanged, regressed (pass to fail), improved (fail to pass). Count them by tag. A change that regresses two billing rows and improves four shipping rows is not "net positive," it is a shipping fix that broke billing, and the billing rows have to be fixed before the change ships. The threshold we use, an editorial rule rather than a vendor's: zero regressions on rows tagged compliance, pricing or handoff, and no more than one regression per ten rows elsewhere, and that one goes into triage before publish rather than after.

Write the numbers down where the next run can see them. A regression set without a results log is a checklist; the log is what makes it a regression set.

Step 5. Triage with a reason, not a rating

A fail is not a fix until you know which kind of fail it is. Intercom's Batch test has a taxonomy already, and it transfers to any platform: when you rate an answer Poor, you choose one of seven reasons, and the help article pairs each reason with indicators; the fixes column below is ours, drawn from the article's Improve this answer recommendations. Rewritten as a triage table for any bot:

Two rules from the source article are worth keeping verbatim in spirit. Ratings "don't train Fin directly," so the fix is always an edit to content, guidance or flow followed by a re-run, never a rating alone. And "Other" gets a written note, because a reason nobody recorded is a regression nobody can find next month.

Step 6. Cadence and ownership

The procedure above takes 45 to 90 minutes on a 40-row set depending on how much of the grading is manual. What keeps it happening is a schedule that does not depend on remembering.

The last row is the one that keeps the set golden. A set that only grows accumulates rows nobody can grade; a set that is never refreshed tests last year's policy. The golden dataset entry has the five maintenance rules.

What our fifteen reviews record

None of our fifteen platform reviews uses the words regression, golden set or batch test (grep -liE 'regression', grep -liE 'golden (set|dataset|test)' and grep -liE 'batch test' each return 0 of 15), which is the reviews' scope, a first-week hands-on, not the vendors' feature set. Three describe the vendor's test console (Botpress, Chatbase, Tidio); grep -liE 'emulator|playground' returns 5 of 15 because it also matches Voiceflow once in passing and Chatfuel on an unrelated sense of the word. The Intercom review (last updated 28 May 2026) does not cover Batch test; that is on the list for its next refresh, and this guide is the place the feature is documented until then.

Where it breaks

The set is the knowledge base. Covered in step 2; the symptom is a set that never fails.

The judge is wrong. An LLM judge, whether Voiceflow's Evaluation or Intercom's Simulation, reads against criteria you wrote, and a vague criterion produces a confident wrong verdict. Voiceflow's own example is the fix: not "rate customer satisfaction" but "rate how satisfied the customer appears based on their tone, whether their questions were answered, and whether they expressed frustration or gratitude." Grade ten rows by hand, run the judge on the same ten, and rewrite the criterion until it agrees on eight. Intercom adds a second trap: success criteria that require an intermediate step (our example: requiring that Fin ask for an order number it already found) fail a Procedure that did the sensible thing; the comparison article's advice is to "focus success criteria on outcomes, not intermediate steps."

The sandbox passed and production failed. A data connector returned its sample payload in the batch run and the real API returned an error in front of a customer. The integration rows in the set are Preview-mode rows against a test user, and the log should say so.

Someone tested on the live bot. Preview conversations land in Intercom's inbox and are excluded from reporting; on platforms without that exclusion, a 40-row run inflates the day's containment and conversation counts. Use a test user, tag the conversations, and exclude them from the month's numbers.

Two changes shipped together. The vendor's model upgrade landed the same afternoon as your prompt edit and the diff has two suspects. Roll back the one you control, re-run, and you have your answer in one more run.

FAQ

How is regression testing different from the QA checklist before launch?

The launch checklist is run once, in an afternoon, before customers see the bot; ours is at chatbot QA testing protocol. Regression testing is the same frozen set of questions run before and after every later change, with the results kept, so that a change to one thing is caught when it breaks another. The first has no test set between runs; the second is nothing but the test set. The protocol page's after-launch note (re-run Tests 1-3 and 9 on the changed paths for a content update) is the minimum for a bot without a test set; once you have the set, step 1 of this guide replaces it.

How many test questions does a chatbot regression set need?

Thirty to fifty for a bot with five to ten topics, three to five per topic, with about a quarter of the rows that should refuse, hand off or ask a clarifying question. Intercom caps a Batch test group at 50; keep more groups if you need them. Fifteen, which is what our reviews use for a first-week check, finds a failure class but is too few to see a regression by topic.

Can chatbot testing be automated?

Partly, and more on some platforms than others as of 7 September 2026. Intercom's Batch test answers up to 50 questions at once but you grade them; its Simulations return an automated pass or fail, for Procedures only. Voiceflow's Tests run scripted conversations with Exact-response, LLM-as-judge, routing and tool-call checks and return a pass or fail per check; its Evaluations score transcripts against criteria you write. Botpress, Chatbase and Tidio give you a console to type into. On every platform the expected outcome per row is written by a person.

Does the vendor's test tool cost anything?

Read 7 September 2026: Intercom's Batch test is "free to use" and Simulations are included with Procedures with a monthly run limit of 250 to 12,500; Botpress's Emulator "does not count towards usage quotas" per the Academy lesson, though the vendor FAQ cited in our review says the opposite, so check your usage panel; Voiceflow's Tests consume credits "the same as in a real conversation" and its Evaluations per transcript by model and length; Chatbase's Playground consumes message credits per instance, doubled in Compare Mode per our review; Tidio does not state whether Playground questions count toward the 50-conversation free quota, so check usage after a run.

What should I do when the vendor upgrades the model underneath my bot?

Treat it as a change you did not make: run the full set. If the platform lets you compare models side by side (Chatbase's Compare Mode does), run the set on both before switching. Vendors ship changes without asking, which is why a monthly full run with no change of your own is in the cadence table.

Related guides

• Golden dataset — the anatomy of a row, where rows come from, and the five maintenance rules.
• Chatbot QA testing protocol — the ten one-off tests before first launch.
• Reduce chatbot hallucinations — the weekly production check that feeds new rows into the set.
• AI red-teaming guide — how to write the adversarial quarter of the set.
• Build a chatbot knowledge base — the content most changes touch.
• Prompt engineering for chatbots — the other thing most changes touch.
• Chatbot confidence policy — what the refusal and clarification rows are testing.
• Chatbot prompt injection tester — a source of adversarial rows.

Sources

• Intercom Help, Batch test Fin AI Agent (article 10521711, written by Beth-Ann Sher, dated 13 May 2026), read 7 September 2026: the question sources (past conversations, AI topic, manual, CSV with "a single column containing up to 50 test questions"); "up to 50 questions based on your most recent customer conversations (30-90 days)"; the compliance-critical edge-case sentence; the Test-as options and brand selection; the seven Poor reasons with their indicators and the recommendations under "Improve this answer"; "These ratings don't train Fin directly"; the test-group settings sentence quoted in step 4; the reset-versus-re-run FAQ; the yellow language-settings messages; "the Fin AI Agent > Test page is free to use"; Fin Vision not supported; data connectors "use their configured test or sample payloads"; Tasks, Procedures and Workflows "are not executed."
• Intercom Help, Simulations vs. Batch tests vs. Previews (article 14077180, written by Dawn, dated 31 July 2026), read 7 September 2026: the quick-reference table (testing mode, "Creates a real conversation?", "Hits live APIs?", saves and reruns, pass/fail, usage limits "250–12,500/month"); "Catching regressions after Procedure updates" under Simulations' best-for; "Running Simulations before every Procedure launch is strongly recommended"; "Preview conversations appear in the inbox but are excluded from reporting"; "If you need to test with real API calls, use Preview and select a specific user with real data in their external system"; "Focus success criteria on outcomes, not intermediate steps."
• Botpress Academy, Guide to the Botpress Interface: Studio — Emulator (botpress.com/academy-lesson/studio-ui-emulator), read 7 September 2026: the ↑ and ↓ resend keys; the in-line logs; the inspect window for Autonomous Nodes; "it does not save conversations or count towards usage quotas. For production-like testing, use the Share button or deploy on the intended channel." Our Botpress review cites the vendor's FAQ as counting emulator conversations; the two vendor pages disagree, the Academy lesson is quoted here, and the review is flagged for its next refresh.
• Voiceflow Documentation, Evaluations (voiceflow.com/docs/documentation/measure/evaluations), read 7 September 2026: the four metric types; the criteria example quoted in Where it breaks; "Test on last transcript"; "Batch run evaluation" on selected transcripts; the three default evaluations; per-evaluation credit cost by model and transcript length. Voiceflow Documentation, Tests (voiceflow.com/docs/documentation/measure/tests), read 7 September 2026 in a browser: the key-concepts table (Test, Turn as User / Agent / Simulation, Check, Persona, Past run); the six-step best practice ("Follow this workflow for any meaningful change to your agent": Clone Main, Make your change, Add tests if needed, Run your tests, Iterate, Ship the change) and the sentence "Existing tests catch behaviour you didn't mean to change"; the three check types (Response with Exact response or LLM as judge, Routing, Tool call); "Tests never run automatically. You always start them yourself"; Run all tests, parallel runs, Retry failed, Past runs; "Tests consume credits when LLM-powered features are used, the same as in a real conversation."
• Chatbase, Introducing Compare Mode in the Playground (changelog, 12 August 2024), read 7 September 2026: "compare the performance of various models, temperatures, and instructions to determine which configuration best suits your specific use case before deploying your chatbot." Chatbotscape, Chatbase review (/reviews/chatbase-review): "compare-models doubling per-query consumption."
• Tidio Help Center, Lyro - a quick setup (article 15607494952604, dated 22 July 2026), read 7 September 2026: the Playground step and the refine step. Tidio Help Center, Lyro - the conversational AI agent (article 9003475527196, updated 17 July 2026), read 7 September 2026: the Playground's Live chat and Email tabs ("test Lyro's live chat capabilities as well as its email responses"); the Test Lyro button, "only for live chat, however - there is no email testing option this way"; the Add answer button "directly inside the test widget." Chatbotscape, Tidio review (/reviews/tidio-review), the Scenario D knowledge-base section: the 15-question Playground run and the "first-week snapshot" sentence.
• Chatbotscape review corpus (the fifteen platform reviews at /reviews), searched 7 September 2026; the search strings and counts are printed in the sources of /glossary/golden-dataset.
• Ahrefs Keywords Explorer, US overview, queried 7 September 2026: the figures in this guide's keyword note.
• Chatbotscape evaluation methodology. /methodology (continuously updated).

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is part of our SMB chatbot Academy and is written for the owner or support lead of a small business whose chatbot is live and keeps being changed. It reads five vendors' testing documentation as worked examples of the harness an operator relies on; it does not rank them, and the platforms named carry affiliate links disclosed above.

Methodology

The two Intercom help articles, the Botpress Academy lesson, the Voiceflow Tests and Evaluations pages, the Chatbase changelog entry and the two Tidio help-center articles were read in full on 7 September 2026, and every quotation carries its source. The corpus counts are our own and the commands are printed in the glossary sibling. The editorial judgments on this page: the trigger table in step 1, the 30-to-50-row size and the one-quarter refusal share, the zero-regression threshold on compliance, pricing and handoff rows and the one-in-ten elsewhere, the 45-to-90-minute run estimate (untimed, from the row count and the manual grading share), and the cadence table. Intercom's reason codes are reproduced as a taxonomy because they are the only vendor-published one we found; the fixes column is ours.

Last updated

8 September 2026.