LLM Routing· LLM infrastructure
LLM Routing — Choosing the Model Per Question
Quick answer: Most customer support questions are simple, and a few are hard. Sending all of them to your strongest model pays the strongest price for every answer. A router sends the simple ones to a cheaper model and keeps the strong model for the rest. The saving is real only if the router rarely sends a hard question to the cheap model, so a router is judged on its mistakes as much as on its price.
How a router decides
A router has to answer one question for every request: which model? Four approaches are common, and they can be combined.
| Approach | How it decides | Strength | Weakness |
|---|---|---|---|
| Rules | Fixed conditions you write: topic keywords, message length, customer tier, whether a tool is needed | Predictable and free to run | Misses anything you did not anticipate |
| Classifier | A small model or a trained classifier labels the request, for example easy or hard, then a table maps the label to a model | Handles varied wording | Needs labelled examples and adds a step |
| Learned router | A model trained on preference data, such as comparisons of which of two models produced the better answer, predicts whether the cheap model is good enough | Tuned on real comparisons | Trained on someone else's questions, not yours |
| Model as router | A small language model reads the request and picks | Easy to start | Costs a call and can be inconsistent |
Whatever the method, the output is the same: a model name for this request. Many routers also let you set a cost-quality threshold, so you decide how often the strong model is used.
Routing, fallback and gateway are different things
These three terms are often mixed up.
- Routing chooses a model before the call, based on the request, to balance cost and quality.
- Fallback switches to another model after a call fails, for example when a provider is down. It reacts to errors, not to the content of the question.
- An LLM gateway is the wider layer in front of the providers that handles keys, limits, logs, retries and often both routing and fallback.
A chatbot can use any one without the others. A router with no fallback has no plan for an outage, and a gateway with no routing sends every question to the model you named.
What the documentation says about three options
These are project statements, not our test results, and we have not benchmarked any of them.
- RouteLLM. An open-source framework from lm-sys for serving and evaluating routers, under the Apache-2.0 license. You name a stronger model and a weaker one, and the router decides per query. Its README lists several shipped routers: a matrix-factorization router (the one it recommends for most uses), a weighted-Elo ranking router, a BERT classifier, a language-model classifier and a random router. A per-request threshold sets the trade-off, and a calibration tool picks a threshold for the share of queries you want sent to the strong model. The README claims costs reduced by up to 85 percent while keeping 95 percent of GPT-4 performance on widely used benchmarks such as MT Bench. That is the project's claim on its benchmark, not a result on a support chatbot.
- LiteLLM auto routing. LiteLLM's documentation describes a single router that classifies each request, then sends it to a target for its tier, which can be one pinned model or a pool. The page lists heuristics, a language-model classifier, keyword rules and a custom classifier plugin as ways to classify. The page says auto routing is in beta, so configuration keys and defaults can change between releases.
- OpenRouter Auto Router. A hosted router. Its documentation says a lightweight classifier assigns the prompt one of about 30 task types, then the router ranks models for that task by their share of OpenRouter spend over a trailing seven-day window, with a cost tier setting that selects a price band. You can restrict the candidates with an allowed-models list, and the response reports which model actually answered. The documentation says there is no additional fee for the router, and you pay the selected model's standard rate.
Note that these three choose differently: RouteLLM predicts from preference data, LiteLLM classifies by tier, and OpenRouter ranks by popularity within a task type. "Popular" and "good for your customers" are not the same thing.
When a chatbot benefits from routing
Routing earns its place when three things are true: the question mix is uneven, with a large share of simple questions; a cheaper model passes your own test on those questions, as in testing a smaller model; and your volume is high enough that the saving matters. A chatbot with a handful of conversations a day gains little. A chatbot whose questions are all hard, such as regulated advice, gains little too, because almost nothing can safely go to the cheap model.
Routing also pairs naturally with a small language model on the cheap side. If the small model is self-hosted, the router decides which questions use that capacity.
Limits and risks to plan for
- A wrong choice is a wrong answer. When the router sends a hard question to the cheap model, the customer gets a weaker answer, and the cost can exceed the saving. Measure the mistakes, not just the average.
- Public benchmarks are not your questions. A router trained or tuned on general data may sort your product's questions badly. Test it on your own golden dataset and score it with a method such as LLM-as-a-judge, with a human check.
- The router is a step in the path. It adds a decision, and sometimes a model call, to every answer. Measure the added delay.
- Behavior differs between models. Tone, refusals, formatting and tool use change from one model to the next, so a customer can see the style change mid-conversation. Decide whether a conversation sticks to one model once it starts.
- Hallucination risk is not equal. A cheaper model may invent more. Keep an eye on AI hallucination, especially for answers that quote policy, prices or dates.
- Routers drift. Your question mix and the models both change. A router that passed in the spring needs a new test after a model update, which is where LLM evaluation becomes a routine, not a one-off.
FAQ
What is LLM routing?
Choosing which language model answers each request, instead of using one model for everything. A router picks a cheaper model for easy requests and a stronger one for hard requests, or picks by topic, customer tier or task.
Is LLM routing the same as model fallback?
No. Routing decides before the call, based on the request. Fallback switches to a backup after a call fails. They solve different problems and are often used together.
Does LLM routing save money?
It can, when many requests are simple and a cheaper model handles them correctly. The saving is cancelled if wrong routes lead to more complaints or human handoffs, so the net effect needs to be measured on your own questions.
What is the difference between an LLM router and an LLM gateway?
A router chooses the model. A gateway is the wider service that also handles keys, budgets, retries, fallbacks and logs, and it often contains a router.
What are the main types of LLM router?
Rule-based routers, classifier-based routers, routers trained on preference data and a small model acting as the router. RouteLLM, LiteLLM and OpenRouter each implement a different mix.
How do I know if my router is hurting answer quality?
Run the same set of real questions through the routed setup and through the strong model alone, score both, and compare. The companion guide shows the steps.
Related terms
- LLM gateway — the layer that often contains the router.
- Small language model — a typical cheap destination.
- Model distillation — another way to get a cheaper model.
- Large language model — the strong destination.
- LLM evaluation — how routing decisions are tested.
- LLM-as-a-judge and golden dataset — the scoring method and the test set.
Sources
- lm-sys, RouteLLM, GitHub repository — github.com/lm-sys/RouteLLM, read 17 October 2026. Source of the router list, the threshold description, the license and the 85 percent and 95 percent claims (project claims).
- LiteLLM, Auto routing (beta) — docs.litellm.ai/docs/proxy/auto_routing, read 17 October 2026.
- OpenRouter, Auto Router — github.com/OpenRouterTeam/docs, documentation source read 17 October 2026.
- Chatbotscape evaluation methodology. /methodology (continuously updated).