Zero-Shot vs Training Data

The Four Rungs, and How to Know Which One You Are On (2026)

Quick answer: Start zero-shot, because it costs nothing and you have no examples to collect on day one anyway. Then build a test set of thirty real messages with the correct category written next to each, which is the one piece of labeling zero-shot does not remove. Read the errors that test set produces, and climb only the rung those errors point at. Most teams skip the free rung, which is rewriting the category descriptions, and go straight to buying examples or a training run. That is the expensive mistake, and it is expensive twice, because a classifier trained on a boundary you never managed to state will reproduce your ambiguity faster and with more confidence.

Our glossary entry defines the term, separates its two research senses, and explains why the label text becomes the specification once the examples are gone. It stops before the decision. This page picks it up there, and it assumes you have a bot that classifies something and a nagging suspicion that it should be doing better.

If your question is instead which data to gather and how to curate it once you have decided you need some, our training data guide is the right page. This one is about whether you need any yet.

Step 1: build the test set before you touch anything

Zero-shot took the training set off your plate and left the test set exactly where it was. This is the least popular sentence in the category and the most load-bearing.

Here is why it holds. Without examples, your classifier is a piece of writing: category names and descriptions that state what belongs where. There is no way to know whether a piece of writing is any good except by running things through it. And there is no way to know whether an edit helped except by running the same things through it again. The test set is not a QA formality bolted on at the end. It is the only instrument you have.

Build it like this, and build it first:

• Thirty real messages, pulled from your support inbox, your chat transcripts, or your contact form. Not invented ones. Invented messages are written by someone who already knows the categories, which is exactly the knowledge the customer does not have.
• The correct category next to each, decided by a person with the authority to decide. Where two people disagree about a message, that disagreement is your finding, and it usually means two categories share a boundary nobody has drawn.
• A handful of deliberate out-of-scope messages, five or so, that belong in no category at all. A bot that has never been shown a message it should refuse has not been tested.
• Per language, if you sell in more than one. Accuracy does not transfer, and our multilingual guide covers why assuming it does is the most common way a second market goes quietly wrong.

Thirty is a working floor rather than a statistical claim. It is small enough to grade by hand in under an hour and large enough that a bad edit shows up. Grow it whenever a real conversation surprises you, which is the habit our QA testing protocol builds around.

Step 2: the four rungs

There are exactly four places the specification of a category can live, and they form a ladder because each one costs more than the last and buys something different.

Rung 1: zero-shot with default labels. You name your categories, the model sorts messages into them, and nothing else exists. This is where every platform starts you, and on a narrow bot with well-separated categories it is often where you should stop. Cost: nothing beyond the per-message call.

Rung 2: zero-shot with rewritten labels. Same mechanism, better writing. Each category description now states its own edge, including what the neighboring category covers instead. Cost: an afternoon. This is the rung teams skip, and skipping it is why so many intent projects arrive at a training run that did not need to happen.

Rung 3: few-shot exemplars. You put a handful of worked examples in the prompt itself, so the model sees two or three correctly sorted messages before it sorts yours. Nothing about the model changes and the examples vanish when the request ends. Cost: a longer prompt on every single call, forever, which is a real line item at volume, plus the examples themselves.

Rung 4: a trained classifier or a fine-tune. Now examples become a dataset and the boundary is learned rather than described. This is the classic intent recognition setup, or a fine-tuned model, and both produce an artifact you version, evaluate and re-create when the thing underneath them is deprecated. Cost: a dataset in the hundreds to thousands of examples, an evaluation harness, and a standing maintenance commitment. Our fine-tuning entry covers what that commitment actually involves and why most SMB bots never take it on.

Rung 4 is not the top of a quality ladder. It is a different trade: worse at unanticipated phrasing, better at cost per call, latency, and predictability. Plenty of production systems run rungs 1 and 4 together, with a cheap classifier taking the confident majority and a model handling the rest.

The research literature happens to recommend the same climbing order, for its own reasons. The Kojima paper that introduced zero-shot chain-of-thought closes by urging attention to zero-shot capability "before crafting finetuning datasets or few-shot exemplars," having shown that a single added sentence moved accuracy on one benchmark from 17.7 to 78.7 percent. That figure is about reasoning tasks, not intent labels, so take it as a reason to exhaust the writing before you buy the data rather than as a number that transfers.

Step 3: let the errors pick the rung

Grade your thirty messages, sort the failures by pattern, and the pattern names the rung. Guessing instead is how a team ends up with a fine-tune that fixed nothing.

Two readings of this table are worth making explicit. The first four rows are quality problems and the last two are economics problems, and they are the only rows where rung 4 is the honest first answer. If your bot is cheap enough and fast enough and simply wrong sometimes, the training run is not what you need.

The second reading is that rung 2 appears in five of eight rows. That is not an accident of how we wrote the table. It reflects the structural fact from the glossary hub: once the examples are gone, the description carries the entire specification, so most defects in a zero-shot classifier are defects in a paragraph somebody wrote quickly.

Step 4: rewriting a label, concretely

Since rung 2 is the one that pays and the one that gets skipped, here is what the work actually looks like.

Take the confusable pair first. A bot with a category set of cancel_order, change_order and return_order will scramble them, because the names alone say nothing about the timeline that separates them. The rewrite states the edge and, crucially, names the neighbor:

• cancel_order: the customer wants the order stopped and nothing sent. Applies only before dispatch. If the order has shipped or arrived, this is return_order instead.
• change_order: the customer wants the order to continue in a modified form, different size, address, quantity. Applies only before dispatch. If they want it stopped entirely, this is cancel_order.
• return_order: the goods have arrived and the customer wants to send them back. Anything after delivery lives here regardless of how the customer phrases it.

Three things are doing the work. Each description names the discriminating fact, which here is dispatch. Each names its neighbor and the condition that hands over to it. And each anticipates the phrasing collision, which is that customers say "cancel" about deliveries that already arrived, constantly.

Then re-run the thirty messages. If the pair stops swapping, you are done and you spent an afternoon. If it does not, you have learned something more useful than a fix: the boundary you think exists may not exist in your business, and no amount of machine learning will invent one. Merging the two categories is a legitimate outcome here, and it is usually the right one when your own team cannot agree on which side a message falls.

A note on scope while you are in there. The intent recognition entry recommends keeping the set small, in the range of five to fifteen categories per domain, and that guidance holds harder under zero-shot than under a trained classifier. Every category you add is another boundary you have to state against every neighbor, and the number of boundaries grows faster than the number of categories.

Step 5: the two things that never move down the ladder

Whatever rung you land on, two things stay.

The out-of-scope category. A classifier with no way to say "none of these" will always name the closest option, and under zero-shot it will do so fluently. That is the mechanism behind most confident misrouting, and the fix is an explicit category plus a threshold beneath it, which the fallback intent entry covers and our fallback-rate guide sizes. A falling fallback rate is not automatically good news, since it looks identical to a bot that has stopped admitting it does not know.

The regression run. Zero-shot accuracy is inherited from a model the vendor can change under you, so a number measured in March is a number about March. Re-run the test set on a schedule and after every label edit, model change, or platform update. This is cheap, it is boring, and it is the only way anyone finds out that an upgrade cost them four points on Spanish.

Step 6: the cost shape, which changes at the rung you least expect

The arithmetic here is shaped rather than numeric, because unit prices differ by vendor and by month, and we would rather give you the shape than a figure that ages badly.

Rungs 1 through 3 all bill per message, and rung 3 bills slightly more per message than rung 1 because the prompt is longer on every call. Rung 4 moves most of the cost to the front, into building and maintaining a dataset, and then runs very cheaply per message. So the crossover is a volume question and nothing else. At low volume, per-message pricing on an already-configured model is almost always cheaper than the labor of a dataset. At high volume, that reverses, and it reverses at a threshold you can compute from your own numbers rather than infer from a blog post.

The honest way to run that calculation is with your vendor's actual metered unit in front of you, since platforms package this very differently and the unit is often not tokens. Our pricing models guide covers the packaging patterns, and the ROI quick math is where the sum belongs. The number that surprises teams is rarely the model cost. It is the maintenance cost of rung 4, which is a standing commitment rather than a project.

Step 7: five situations and what each one calls for

If you are launching and have no historical transcripts, stay on rung 1 and spend the time building the test set instead. You cannot collect training examples for phrasings you have never received, and thirty labeled messages from your support inbox will teach you more in an hour than a training set assembled from guesses.

If a specific pair of categories is producing most of your errors, go to rung 2 and rewrite only that pair. Resist the urge to rewrite everything at once, because then you will not know which edit moved the number.

If accuracy is acceptable but the format of what comes back keeps drifting, go to rung 3. Exemplars fix format more reliably than instructions do, and you need very few of them.

If your volume has grown to the point where per-message model cost is a visible line on the invoice, and your categories have been stable for months, rung 4 is finally worth pricing. Stability is the condition that matters there, because a dataset built against categories you are still redesigning is work you will do twice.

If your own team cannot agree on the correct category for a message in your test set, stop climbing entirely. The problem is not the classifier. Merge the categories, or write down the rule a human uses, and only then decide what technology should enforce it.

Platform notes: which rungs are actually available to you

Where you can climb to depends on what the product exposes, and the products differ more here than their feature pages suggest. These notes trace to our own reviews at their own dates, between 26 and 31 May 2026, and evidence depth differs per review and per claim.

Botpress is the most flexible of the set for this particular decision, because its review records native routing across OpenAI, Anthropic, Groq and Hugging Face as first-class providers, with different tasks routable to different providers. That means classification can run on a cheaper or faster model than generation, which is a way of buying part of rung 4's economics without leaving the zero-shot mechanism. Its measured figures were produced on a Plus tier with Claude 3.5 Sonnet configured, so they move with that choice by design.

Manychat is the clearest illustration of rung 1 as a product decision. Its trigger sidebar offers to convert a keyword list into an intention directly, which our review notes is exposed at trigger level rather than buried in an advanced tab, so an operator gets classification without first understanding what a training phrase is. That review also flags something worth checking before you budget: two distinct AI products exist and the pricing page blurs them, with a free-tier AI step in the flow builder and a separate paid AI bundle.

Chatbase produced the strongest per-language accuracy in our Tier 1 batch, at 89, 85, 83 and 79 percent across English, Spanish, Brazilian Portuguese and Hindi on a Standard tier, which is a useful ceiling reference for what rung 1 can reach on a well-configured product. Its review also records no BYOLLM support, so the model choice lever that Botpress offers is not available there.

Tidio carries a specific reason to do your own per-language testing rather than trusting a spec sheet: its review records that the vendor does not publish a certified product-level supported-language list with accuracy claims, while the underlying model has broad coverage. That is exactly the gap a test set closes.

Two caveats belong in the open rather than in a footnote. Our Intercom review's intent figures are anchored editorial estimates with paid-tier hands-on validation scheduled rather than complete, and our SendPulse review has no measured per-language accuracy at all, with that pass queued. Do not read those two numbers the way you read the four that were measured. All of these reviews carry affiliate links, and our methodology sets out what that does and does not change about a score.

The question worth putting to any vendor is narrower than "do you support AI." Ask whether you can edit the description of a category and not only its name, and ask what the product does with a message that fits nothing. A vendor whose answer to the second is "it picks the closest one" has told you where your first defect will come from.

The decision, compressed

Build a test set of thirty real messages with correct categories attached, including five that belong nowhere, before you change anything. Run it. Sort the failures by pattern. If adjacent categories are swapping or your vocabulary is being missed, rewrite the descriptions so each names its own edge and its neighbor, then re-run. If phrasing variety or output format is the problem, add a few exemplars to the prompt. Only when your categories have been stable for months and per-message cost or latency has become a real constraint should you price a trained classifier, and price it against the maintenance, not against the build. Keep the out-of-scope category and the regression run whatever you do.

And if the test set comes back clean at rung 1, that is a result rather than a wasted afternoon. It means your categories are well separated and the money you had earmarked for training data should go to the content the bot answers from, which is where it was probably needed anyway.

Frequently asked questions

Related guides

• Zero-shot learning (glossary) — the definition, the two research senses, and why the label text becomes the specification
• Intent recognition (glossary) — the classifier alternative, the small-category-set guidance, and a worked comparison
• Chatbot training data — what to collect and how to curate it, once you have decided you need some
• Chatbot training (glossary) — the four unrelated mechanisms the word covers
• Fine-tuning (glossary) — what rung 4 commits you to, and why most SMB bots skip it
• Chatbot QA testing protocol — the wider testing habit this page's test set belongs to
• Prompt engineering for chatbots — the discipline rung 2 is an instance of
• Reduce chatbot fallback rate — how to read the number that tells you out-of-scope is working
• Multilingual chatbot: seven truths — why per-language testing is not optional

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is part of our SMB chatbot Academy. Its central recommendation is that most teams should stay on the free rung and rewrite their category descriptions rather than buy data, a training run, or a higher AI tier, which runs against our own commercial interest in readers upgrading plans through our links. We have a mild commercial interest in readers choosing platforms through our reviews. This is editorial guidance rather than a consulting engagement. To flag an error, write to editorial@chatbotscape.com.

Methodology

This page contains no new measurements. We did not run a classification study for it. The four-rung ladder, the error-pattern-to-rung table, the label-rewriting worked example, the two things that never move down the ladder, the cost-shape argument and the five situations are our editorial framing rather than a published standard. Every count on the page is a planning anchor rather than a study result, and each is named here so none of them can be mistaken for a finding: the thirty-message test set, the five out-of-scope messages, the two-to-three exemplars per problem category, the tenth-example signal, and the five-to-fifteen category range carried from our intent recognition entry. The cancel_order / change_order / return_order example is illustrative and written for this page. The Kojima figures are quoted from that paper's published abstract and describe reasoning benchmarks, not intent classification; they appear here to argue for an order of operations, not as a transferable accuracy expectation. Platform intent-accuracy figures are stated as of each review's own date and differ in evidence depth, which is disclosed inline: the Chatbase, Manychat, Tidio and Botpress figures are hands-on measurements on twenty-query sets, the Intercom figures are anchored editorial estimates with paid-tier validation scheduled rather than complete, and the SendPulse review carries no measured per-language accuracy at all. Definitions and the two senses of zero-shot are carried from our glossary entry and its cited primary sources rather than re-derived.

Sources

• Kojima, Takeshi, Gu, Shixiang Shane, Reid, Machel, Matsuo, Yutaka and Iwasawa, Yusuke. Large Language Models are Zero-Shot Reasoners. NeurIPS 2022; arXiv:2205.11916, v4 of 29 January 2023. arxiv.org/abs/2205.11916 (abstract fetched and quoted verbatim 30 July 2026). Source of the 17.7 to 78.7 percent figure and of the closing recommendation to examine zero-shot capability "before crafting finetuning datasets or few-shot exemplars," which is the published argument closest to this page's climbing order.
• Brown, Tom B., Mann, Benjamin, Ryder, Nick, et al. Language Models are Few-Shot Learners. arXiv:2005.14165, v4 of 22 July 2020. arxiv.org/abs/2005.14165 (abstract fetched and verified 30 July 2026). Source of the in-context definitions of zero-shot and few-shot that rungs 1 and 3 rest on, and of the observation that conventional fine-tuning "requires task-specific fine-tuning datasets of thousands or tens of thousands of examples," which is the cost rung 4 inherits.
• Chatbotscape Glossary. Zero-shot learning — carries the two research senses, the Palatucci et al. (NeurIPS 2009) classical citation, and the label-as-specification argument inherited by this page (verified 30 July 2026).
• Chatbotscape Glossary. Intent recognition — carries the classifier-versus-LLM comparison, the five-to-fifteen category guidance, and the out-of-scope failure mode referenced in Steps 4 and 5.
• Chatbotscape platform reviews for Botpress, Manychat, Chatbase, Tidio, Intercom and SendPulse, last updated between 26 and 31 May 2026. Platform notes are structural and stated as of each review's own date. Evidence depth differs by product and is recorded per review rather than flattened into a single claim on this page.
• Chatbotscape evaluation methodology. /methodology (continuously updated).

Last updated

31 July 2026 — Initial publication aligned to methodology v3.12.1. Next scheduled refresh: 31 October 2026.