How to Route Chatbot Questions to a Cheaper Model (Without Hurting Answers)

Quick answer: Sort a sample of your real questions into easy and hard, send the easy kind to a cheaper model with a few plain rules, send everything uncertain to the strong model, and then run the same questions through the routed setup and through the strong model alone and compare the scores. In our invented example the model bill falls by $134 a month, but about 22 extra human contacts a month would cancel that saving, which is why the test matters more than the router.

Step 1: Check that routing is worth trying

Routing only pays when a cheaper model can do part of the job. Before building anything, do the groundwork in testing a smaller model: if no cheaper model passes your own test on any group of questions, there is nothing to route to. Then ask two more questions. Is a large share of your traffic simple and repetitive, such as opening hours, order status or password help? Is your monthly model bill big enough that a saving of a third would be noticed? Two clear yeses mean continue. If your bill is small, a simpler fix is in the semantic cache guide or in tighter prompts.

Step 2: Sort a sample of real questions

Take 100 recent customer messages, or 100 test questions if you have no traffic yet. Sort each into one of three piles by hand:

• Easy: one fact, one step, a clear answer in your knowledge base.
• Hard: needs reasoning across several facts, an exception to a policy, a long message with several questions, or a tool call.
• Risky: money, legal wording, safety, an angry customer or an account change. These always go to the strong model or to a person, whatever else the router says.

Write down how many fall in each pile. That ratio is your realistic ceiling: if only 20 of 100 are easy, no router will save much. This sorting is our own practice, and it takes about an hour.

Step 3: Choose a routing method

Match the method to your size, using the types in the LLM routing entry:

1. Start with rules. Keywords, message length, channel and whether a tool is needed are free to run, easy to read and easy to fix. For a small support bot, rules plus a safe default are often enough.
2. Add a classifier later if rules miss too many wordings. A small model labels the request easy, hard or risky, and a table maps the label to a model. It costs one extra call per message.
3. Consider a ready-made router if you want one. RouteLLM is an open-source framework under Apache-2.0 whose README describes routers trained on preference data and a threshold you set. LiteLLM's documentation describes a beta auto router that classifies requests into tiers, and OpenRouter's documentation describes a hosted Auto Router that picks a model by task type. All three are project statements we have not benchmarked. A ready-made router is trained on other people's questions, so it still has to pass your test in Step 5.

If you already use an LLM gateway, check whether it offers routing before adding a second service.

Step 4: Write the first rules and a safe default

Write the rules as a short ordered list. The first match wins.

1. If the message mentions a refund, payment, cancellation, legal wording, a complaint or a safety issue, use the strong model.
2. If the message needs a tool call, such as an order lookup that changes data, use the strong model.
3. If the message is longer than your chosen length, or has more than one question, use the strong model.
4. If the message matches one of your known easy topics and the answer exists in the knowledge base, use the cheap model.
5. Everything else goes to the strong model.

The last line is the safe default, and it is the most important rule. A router that sends unknown cases to the cheap model fails on exactly the questions you did not predict. Start with the cheap model handling only a small, clearly easy slice, and widen the rules only after the test in Step 5 supports it. Decide too whether a conversation sticks to one model once it starts, to avoid the style changing from one reply to the next.

Step 5: Test routed answers against the strong model

This is our own test, and it is the step that proves the setup.

1. Take a test set of at least 100 real or realistic questions, with the correct answer or a checklist for each. If you have no set yet, build a small golden dataset.
2. Run all of them through the strong model alone and save the answers.
3. Run all of them through the router and save the answers, plus which model answered each.
4. Score both sets the same way: a person for the risky and hard piles, and for the rest a model-as-judge check that a person samples. See LLM evaluation for how scoring methods compare.
5. Count three things: how many answers got worse under routing, how many questions the router sent to the cheap model that a person would call hard, and how many invented a fact. Weigh the last one most, because the cheap model may be more prone to AI hallucination.
6. Set a pass bar before you look: for example, no risky question reaches the cheap model, and no more than 2 percent of cheap-routed answers score worse than the strong model's answer.

A set of 100 is a smoke test, not a proof. A small error rate cannot be estimated reliably from 100 questions; the benchmark guide explains how wide the uncertainty is at that size.

Step 6: Release gradually and watch

Send a small share of live traffic through the router first, for example 10 percent, and compare it with the rest on the signals you already track: escalation to a person, repeat questions in the same conversation and customer ratings. Log which model answered each message so any complaint can be traced. Retest after every model update and every time your product changes, because routers drift as the question mix and the models change. The production monitoring guide shows where these signals fit.

Check the arithmetic

Every number below is invented for illustration, not a measurement or a vendor price.

Two lessons. First, the saving is small in absolute terms and is wiped out by a very low error rate: if just 0.5 percent of the cheap-routed conversations, which is 70, led to a human contact, the cost would be $420 and the net result would be about $286 worse than not routing. Second, the rules-only version has no classifier cost, so the saving would be $140 before errors, which shows that the cheapest router is often the simplest one. The rule we use: if you cannot measure mistakes, do not route.

Common mistakes

• Sending unknown questions to the cheap model. Make the strong model the default.
• Trusting a public benchmark instead of your own questions. A router tuned on general data may sort your product badly.
• Skipping the risky pile. Money, legal and safety questions should never depend on a cost decision.
• Judging only the average score. A few serious failures can sit behind a good average.
• Never retesting. A model update can change who is "good enough".
• Routing before fixing the prompt. A clearer prompt sometimes lets the cheap model handle more without any router.

FAQ

How do I route easy questions to a cheaper model?

Sort real questions into easy, hard and risky, write a few ordered rules that send only clearly easy topics to the cheap model, make the strong model the default, and test the result against the strong model alone before release.

Should I use a rule-based router or a classifier?

Start with rules, since they are free, readable and easy to correct. Add a small classifier only if rules miss too many wordings and the extra call is worth its cost.

Can I use RouteLLM, LiteLLM or OpenRouter instead of writing rules?

Yes, each describes a routing option, and each is a project statement we have not benchmarked. Whichever you choose, it needs to pass your own test, because it was not tuned on your questions.

How do I test that routing is not hurting quality?

Run the same questions through the strong model alone and through the router, score both the same way, and compare worse answers, hard questions sent to the cheap model, and invented facts against a pass bar you set first.

How much will routing save?

It depends on your easy share, the price gap between the two models and your error rate. In our invented example the bill falls from $240 to $106 a month, and about 22 extra human contacts would cancel it.

What if the cheap model goes down?

Routing does not cover outages. Add a fallback to a backup model, as in the gateway setup guide.

Related guides

• Reduce chatbot costs guide — all the levers, ranked.
• Test a smaller model for your chatbot — the test that gives a router something to route to.
• Self-host a small model or use an API — where the cheap side can run.
• Set up an LLM gateway — where routing and fallback often live.
• LLM routing, small language model and LLM gateway — the glossary entries behind the terms used here.

Sources

• lm-sys, RouteLLM, GitHub repository — github.com/lm-sys/RouteLLM, read 17 October 2026: shipped routers, threshold, license.
• LiteLLM, Auto routing (beta) — docs.litellm.ai/docs/proxy/auto_routing, read 17 October 2026.
• OpenRouter, Auto Router — github.com/OpenRouterTeam/docs, documentation source read 17 October 2026.
• Chatbotscape evaluation methodology. /methodology (continuously updated).

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from project documentation; the question-sorting step, the safe-default rule, the side-by-side test and the worked example are our own practice with invented numbers, and we have not benchmarked any router.

Last updated

18 October 2026 — first published.