Two Intercom help-center articles (Batch test Fin AI Agent, 13 May 2026; Simulations vs. Batch tests vs. Previews, 31 July 2026), the Botpress Academy lesson on the Studio Emulator, Voiceflow's Tests and Evaluations documentation pages, Chatbase's changelog entry introducing Compare Mode (12 August 2024) and two Tidio help-center articles (Lyro - a quick setup, 22 July 2026; Lyro - the conversational AI agent, 17 July 2026) were read on 7 September 2026, and every limit, count and quotation below carries that date; they change without notice. Every count of our own reviews is a tally of the fifteen files matched by sample-reviews/*-review.md, with the search string printed in Sources.

Golden dataset· Testing and evaluation

A golden dataset (also golden set, golden test set) is a fixed collection of inputs, each paired with the outcome a human has confirmed to be correct, kept separate from anything the system learns from and used to check the system's behavior. For a chatbot, a row is a customer message plus the answer, action or handoff it should produce, tagged with the topic, language and channel it tests. The set is run before and after a change to the knowledge base, the prompt, the flow or the model, and the difference between the two runs is the regression report.

By Chatbotscape Editorial· Methodology· Published 8 September 2026· Updated 8 September 2026

Golden Dataset — What It Is, How to Build One for a Chatbot, and What the Vendors' Test Tools Accept

Quick answer: A golden dataset is the list of questions you already know the right answer to, written down once and run every time the bot changes. It is "golden" because a person, not the model, decided what correct looks like for each row, and because the rows do not move between runs. The word comes from software testing, where a golden file is the reference output a test compares against; the chatbot version pairs an input with an expected outcome rather than expected wording, because two runs of a large language model rarely produce the same sentence.

The vendors have built the harness for you, with limits worth knowing before you build the set. Read on 7 September 2026, Intercom's Batch test takes up to 50 questions per group from a CSV, past conversations or a paste, returns Fin's answer with the sources it used, and "is free to use"; its Simulations run an AI customer against a Procedure and return a pass or fail, at 250 to 12,500 runs a month depending on conversation volume. Botpress's Studio Emulator "does not save conversations or count towards usage quotas." Voiceflow's Tests run scripted conversations with pass-or-fail checks on the reply, the routing and the tool calls, and its Evaluations score every transcript with a model against criteria you write. Chatbase's Playground runs the same query against several configurations side by side. Tidio's Lyro Playground answers test questions and lets you add the missing answer from inside the test widget. None of them writes the expected outcome for you; that column is the work.

What this entry is and is not

Limits, prices and quotations are read from the vendors' own pages named in Sources, each dated. The five platforms named here all have Chatbotscape reviews that carry affiliate links; see our affiliate disclosure. This entry owns the artifact: what a row contains, where rows come from, what each vendor's tool accepts and returns. The row counts, the hour-per-fifty-rows and 45-to-90-minute run estimates, the one-quarter refusal share and the eight-of-ten judge threshold are editorial rules of thumb, untimed, not vendor figures. The procedure for running the set before and after a change is the chatbot regression testing guide; the ten one-off tests to run before a first launch are the QA testing protocol; the data a bot learns from, as opposed to the data it is checked against, is chatbot training.

What it is not

Three neighbors get called a golden dataset and are not one.

Training data is what the model learns from. The golden set is what it is checked against, and the two must not overlap: a bot that has seen the test question in training will pass the test without being able to answer the paraphrase a customer sends. Intercom draws the line in its own product: the ratings you give in Batch test "don't train Fin directly," and the help article repeats it in the FAQ, "Batch test is strictly for quality assurance." If you find a wrong answer, the fix goes into the content or guidance; the test row stays where it is.

Transcripts are what happened. They are the best source of rows (below), but a transcript has no expected-outcome column; it records what the bot said, which may be the thing you are trying to catch. Voiceflow's Evaluations run over transcripts and score them, which is monitoring; the golden set is the subset you have read, corrected and frozen.

A benchmark is a golden set someone else built for a model in general. The "llm evaluation" results a vendor publishes tell you how a model performed on questions that are not yours, in a domain that is not yours. Useful for choosing a model; useless for knowing whether your refund policy is answered correctly after Tuesday's edit.

Anatomy of a row

A row that can be re-run by someone who did not write it carries seven fields. The first two are the minimum; the other five are what makes the set survive a quarter.

Field	What goes in it	Why it is there
Input	The customer message, verbatim, including the typo or the second language if that is what is being tested	The bot is tested on what customers send, not on what the content author would have typed
Expected outcome	What must be true of the response: a fact it must contain, an action it must trigger, a handoff it must offer, a refusal it must give	Outcome, not wording; Intercom's guidance for Simulations is to "focus success criteria on outcomes, not intermediate steps"
Must not	Anything whose presence is a fail: a price from the old plan, a competitor's name, a promise the policy does not make	Most hallucinations are additions, and a must-contain check does not catch an addition
Tags	Topic, intent, language, channel, user segment	So a failure can be counted by area ("four of the five billing rows regressed") rather than as a total
Source of truth	The article, policy page or ticket the expected outcome was taken from, with the date	When the source changes, the row is due for review; without this field nobody knows which rows a content edit affects
Owner and date	Who confirmed the expected outcome and when	A row nobody owns becomes a row nobody trusts
Last result	Pass, fail or improved, with the run date	The set is only a regression set if it remembers the last run

Intercom's Batch test keeps the input and the last result for you (the question, Fin's answer, a Good / Acceptable / Poor rating, plus a note on Acceptable or a reason on Poor) and exports a CSV "compiling all questions, answers, user-applied ratings, and the sources utilized for each response." Voiceflow's Tests keep the input, the checks and past runs. The expected outcome, the must-not, the tags, the source of truth and the owner live in your spreadsheet, not in the tool.

Where the rows come from, and how many

Four sources, in the order they pay off.

Past conversations give the rows that match what customers ask. Intercom generates "up to 50 questions based on your most recent customer conversations (30-90 days)" and, if AI Topics are on, by topic, which the help article lists as a use case, "prioritizing high-volume or low-CSAT topics." On any platform, the transcript export and the fallback log are the same source by hand.

Tickets the bot did not handle give the rows the bot should have handled. A month of escalations, deduplicated by intent, is usually twenty to thirty distinct questions.

Manual edge cases give the rows no transcript contains yet: the new feature, the policy that changes next week, the compliance question that has been asked once. Intercom's article names this case exactly ("compliance-critical scenarios and/or edge cases that don't appear in historical conversation data") under the manual path; the CSV path is for bulk-loading a curated list.

Adversarial rows give the must-not column something to catch: prompt injection attempts, requests for data the bot must not disclose, the competitor comparison the bot must decline. The red-teaming guide is the method; the golden set is where the survivors are kept.

On size: Intercom's cap is 50 per test group, with as many groups as you want. Our own reviews use 15 questions per platform for the first-week knowledge base check, which is enough to find the failure class and not enough to measure a rate; the Tidio review's 15-question Playground run returned 11 grounded answers, 3 correct refusals and 1 conflated answer, and says in the same paragraph that this is "a first-week snapshot, not a steady-state measurement." For an SMB bot with five to ten topics, 30 to 50 rows with three to five per topic is the size at which a regression is visible by topic and the run fits in an hour on a batch tool, up to 90 minutes on a console.

What each vendor's tool accepts and returns

Platform, tool	Input it takes	Verdict it returns	Limit or cost, as read 7 September 2026
Intercom, Batch test	Up to 50 questions per group from past conversations, an AI topic, a paste or a single-column CSV; run as a user, lead, audience or generic preview user; per brand	Fin's answer with personality, guidance, content sources and triggered automations; you rate Good / Acceptable / Poor and, on Poor, choose one of seven reasons; CSV report	"Free to use"; full seat and All-conversations permission required; Fin Vision not testable; data connectors return sample payloads, no live API calls
Intercom, Simulations	A Procedure plus success criteria; AI plays the customer	Pass or Fail "with full reasoning"	250 to 12,500 runs a month by conversation volume; Procedures only; no live APIs
Intercom, Previews	Typed messages in the training panel	The response and an event log	No limit; creates an inbox conversation "excluded from reporting"; the only mode that hits live APIs
Botpress, Studio Emulator	Typed messages; ↑ and ↓ resend earlier ones; Cmd/Ctrl+E toggles it	The response, in-line logs (variable captured, waiting for input) and an inspect window for each LLM decision	"Does not save conversations or count towards usage quotas" per the Academy lesson; the vendor's FAQ, as cited in our review, says emulator conversations count, so check the usage panel after a run; for production-like testing the lesson points to the Share button or a real channel
Voiceflow, Tests	Scripted conversations from User turns, Agent turns and AI-played Simulation sections, started from a saved persona; Run all tests or a selection; Past runs kept	Pass or fail per check with reasoning: Response (Exact response or LLM as judge), Routing, Tool call	Credits "the same as in a real conversation"; "Tests never run automatically"
Voiceflow, Evaluations	Criteria you write plus a model; runs on every new transcript and, by batch, on selected past ones	Rating (a scale you set), Binary (pass/fail), Options (categories) or Text; three defaults ship (satisfaction 1–5, deflection, resolution in five outcomes)	Credits per evaluation, by model and transcript length; "Test on last transcript" before enabling
Chatbase, Playground Compare Mode	The same query to several instances with different models, temperatures or instructions	Side-by-side responses; no automated judgment	Message credits per instance; our review recorded compare mode "doubling per-query consumption" on the Free tier
Tidio, Lyro Playground	Typed questions on a Live chat tab or an Email tab; a Test Lyro button in the Lyro section opens a live-chat-only test widget	The answer; an Add answer button inside the test widget for a question Lyro could not answer	Not stated on Tidio's Playground or quota pages whether test questions count toward the 50-conversation free quota; check usage after a run. The Test Lyro shortcut has "no email testing option"; the Playground's Email tab does

Two things the table makes visible. Only Intercom's Simulations and Voiceflow's Tests and Evaluations return a verdict without a person reading the answer, and apart from Voiceflow's Exact-response, Routing and Tool-call checks they are LLM judges reading against criteria you wrote, so the quality of the verdict is the quality of the criteria. And only Intercom's Previews touch live systems; every other row is a sandbox, which is what you want for a regression run and not what you want for the last check before a Procedure that calls Shopify goes live.

How the verdict is decided

Exact-match comparison, the golden-file technique this term came from, fails on a chatbot because the wording moves between runs. Three grading methods work, in rising order of cost.

Human rating against the expected outcome. Intercom's Good / Acceptable / Poor with a reason code ("Didn't use the correct content," "Used the content incorrectly," "Tone wasn't right" and four more) is a ready-made taxonomy; the reason is what makes the rating actionable; the article pairs each reason with indicators, and its Improve this answer step with the fix it usually needs. Fifty rows take a reviewer 45 to 90 minutes.

Rule checks. Must-contain and must-not strings, a triggered action, a handoff offered, a response under a length: cheap, deterministic, and blind to meaning. Good for the must-not column, bad for "did it answer the question."

LLM-as-judge. A second model reads the response against the criteria. Voiceflow's LLM-as-judge Response check and Binary Evaluation metric ("Did the agent resolve the issue?") and Intercom's Simulation judge are this. Write the criterion the way Voiceflow's documentation shows, specific rather than general: not "rate satisfaction" but "rate how satisfied the customer appears based on their tone, whether their questions were answered, and whether they expressed frustration or gratitude." Then test the criterion on a transcript you have already graded by hand, which is what the "Test on last transcript" button is for. A judge that disagrees with you on the rows you graded is not ready to grade the rows you did not.

Five rules for keeping the set golden

Freeze it per run, version it between runs. A row edited during a run makes the before-and-after diff meaningless. Change rows between runs, note the change, and re-baseline.
Never let a row into the knowledge base. The moment the test question is also a FAQ entry, the row tests retrieval of itself. Keep the paraphrase in the set and the canonical wording in the content.
Retire rows on purpose. When the policy changes, the row's source of truth is stale; update the expected outcome or retire the row, and record which. A set that only grows accumulates rows nobody can grade.
Balance the columns. A set that is all happy-path passes at 100 percent and tells you nothing. Aim for a quarter of rows that should refuse, hand off or ask a clarifying question; Intercom's reason list has "Didn't clarify the customer's question" as its own failure class.
Grade the judge before trusting it. Whether the judge is a colleague or a model, run it on ten rows you graded yourself and compare. Agreement under eight of ten means the criteria, not the bot, need work.

What our fifteen reviews record about golden datasets

None of our fifteen platform reviews uses the term: grep -liE 'golden (set|dataset|test)' returns 0 of 15, grep -liE 'regression' returns 0 of 15 and grep -liE 'batch test' returns 0 of 15, which says more about the reviews' scope (first-week hands-on) than about the vendors. Three describe the vendor's test console (Botpress, Chatbase, Tidio); the same grep also matches Voiceflow once in passing and Chatfuel on "an earlier playground" meaning a prior test account, not a vendor console, so grep -liE 'emulator|playground' returns 5 of 15. The Chatbase review records the Compare Mode credit cost; the Tidio review records the 15-question Playground run quoted above; the Botpress review records the emulator's inspect window on an AutonomousNode. Intercom's Batch test, the most complete harness in the table, is not in the Intercom review (last updated 28 May 2026), which did not cover the test surface; the review's next refresh should add it.

Chatbot training — the data the model learns from; the golden set must not overlap with it.
AI hallucination — the failure class the must-not column exists to catch.
Chatbot fallback rate — the log that supplies the rows the bot could not answer.
Intent recognition — the tag that lets a regression be counted by topic.
Chatbot knowledge base — the content most changes touch, and the place test rows must not leak into.
Chatbot resolution rate — the production metric; Voiceflow's default Resolution evaluation classifies each transcript into five outcomes.

FAQ

What is a golden dataset?

A fixed set of inputs, each paired with the outcome a person has confirmed is correct, kept apart from anything the system learns from and used to check the system's behavior. For a chatbot, each row is a customer message plus what the reply must contain, do or avoid, tagged by topic, language and channel. It is run before and after a change, and the difference is the regression report.

How big should a chatbot golden dataset be?

Large enough that a regression shows up by topic, small enough to grade in an hour: 30 to 50 rows for a five-to-ten-topic SMB bot, three to five per topic, with about a quarter of rows that should refuse, hand off or ask for clarification. Intercom's Batch test caps a group at 50 questions; you can keep several groups.

Is a golden dataset the same as training data?

No, and they must not overlap. Training data is what the model learns from; the golden set is what it is checked against. Intercom's help center makes the same separation for Batch test: ratings "don't train Fin directly," and fixes go into content or guidance, not into the test.

Can an LLM grade the golden dataset for me?

Partly. Voiceflow's Evaluations and Intercom's Simulations both use a model to judge a conversation against criteria you write, and Voiceflow lets you test a criterion on the last transcript before enabling it. Grade ten rows by hand first and compare; if the judge agrees on fewer than eight, rewrite the criterion before trusting it on the rest.

Which chatbot platforms have a built-in test harness?

Read 7 September 2026: Intercom (Batch test, Simulations, Previews), Botpress (Studio Emulator), Voiceflow (Tests and Evaluations), Chatbase (Playground with Compare Mode) and Tidio (Lyro Playground). Only Intercom's Simulations and Voiceflow's Tests and Evaluations return an automated verdict; the others return the answer for you to grade.

Sources

Intercom Help, Batch test Fin AI Agent (article 10521711, written by Beth-Ann Sher, dated 13 May 2026), read 7 September 2026: the three question sources and the CSV path ("a .csv file with a single column containing up to 50 test questions"); "up to 50 questions based on your most recent customer conversations (30-90 days)"; the compliance-critical edge-case sentence; the Test-as options (User or Lead, Audience, Preview User) and brand selection; the Evaluate answer panel (personality, guidance, content sources, automations); the seven Poor reasons and their indicators; "These ratings don't train Fin directly"; the CSV report sentence; the FAQ answers "the Fin AI Agent > Test page is free to use," "Batch test is strictly for quality assurance," Fin Vision not supported, data connectors "use their configured test or sample payloads," and Tasks, Procedures and Workflows "are not executed."
Intercom Help, Simulations vs. Batch tests vs. Previews (article 14077180, written by Dawn, dated 31 July 2026), read 7 September 2026: the quick-reference table (what each is, testing mode, "Creates a real conversation?", "Hits live APIs?", saves and reruns, pass/fail judgment, usage limits "250–12,500/month," availability); "Preview conversations appear in your inbox, but are excluded from reporting"; "Only Preview can hit live APIs"; "Focus success criteria on outcomes, not intermediate steps."
Botpress Academy, Studio Emulator lesson (botpress.com/academy-lesson/studio-ui-emulator), read 7 September 2026: "You can toggle the Emulator on or off using Cmd/Ctrl + E. Use the ↑ and ↓ keys to navigate and resend previously sent messages"; the in-line logs sentence; the inspect window for Autonomous Nodes; "While the Emulator is ideal for quick debugging, it does not save conversations or count towards usage quotas. For production-like testing, use the Share button or deploy on the intended channel." Our Botpress review cites the vendor's FAQ as counting emulator conversations; the two vendor pages disagree, the Academy lesson is quoted here, and the review is flagged for its next refresh.
Voiceflow Documentation, Evaluations (voiceflow.com/docs/documentation/measure/evaluations), read 7 September 2026: the four metric types (Rating, Binary, Options, Text); the criteria example quoted in the body; "click Test on last transcript"; the three default evaluations and the five Resolution outcomes; "Batch run evaluation" on selected transcripts; per-evaluation credit cost "depends on the model you select and the length of the transcript." Voiceflow Documentation, Tests (voiceflow.com/docs/documentation/measure/tests), read 7 September 2026 in a browser: the key-concepts table (Test, Turn as User / Agent / Simulation, Check, Persona, Past run); the three check types (Response with Exact response or LLM as judge, Routing, Tool call); "Tests never run automatically. You always start them yourself"; Run all tests and Past runs; "Tests consume credits when LLM-powered features are used, the same as in a real conversation."
Chatbase, Introducing Compare Mode in the Playground (changelog, 12 August 2024), read 7 September 2026: "create multiple instances of your chatbot, each powered by different AI models, in a dedicated test environment"; "compare the performance of various models, temperatures, and instructions"; the Dashboard > [Your Chatbot] > Playground path. Chatbotscape, Chatbase review (/reviews/chatbase-review): "7/50 credits in this ~60-minute session" and "compare-models doubling per-query consumption."
Tidio Help Center, Lyro - a quick setup (article 15607494952604, dated 22 July 2026), read 7 September 2026: "In the Playground section, ask questions that customers might submit and review Lyro's answers"; the refine step ("edit, delete, or add responses"). Tidio Help Center, Lyro - the conversational AI agent (article 9003475527196, updated 17 July 2026), read 7 September 2026: the Playground's Live chat and Email tabs ("test Lyro's live chat capabilities as well as its email responses"); the Test Lyro button "in the upper-right" of the Lyro AI Agent section, "only for live chat, however - there is no email testing option this way"; the Add answer button "directly inside the test widget." Chatbotscape, Tidio review (/reviews/tidio-review): the 15-question Playground run (11 grounded, 3 refusals, 1 conflated) and the "first-week snapshot" sentence.
Chatbotscape review corpus (the fifteen platform reviews listed at /reviews), searched 7 September 2026 from the repository root. Denominator: ls sample-reviews/*-review.md | wc -l returns 15. grep -liE 'golden (set|dataset|test)' sample-reviews/*-review.md returns 0; grep -liE 'regression' sample-reviews/*-review.md returns 0; grep -liE 'batch test' sample-reviews/*-review.md returns 0; grep -liE 'emulator|playground' sample-reviews/*-review.md returns 5 (botpress, chatbase, chatfuel, tidio, voiceflow; chatfuel's hits are "an earlier playground" meaning a prior test account, voiceflow's is one passing mention). Passages cited: tidio-review.md line 616; chatbase-review.md line 452; botpress-review.md lines 367 and 599.
Ahrefs Keywords Explorer, US overview, queried 7 September 2026 — the demand, difficulty, CPC, global-volume and parent-topic figures in this entry's keyword note.
Chatbotscape evaluation methodology. /methodology (continuously updated).