Skip to content
Chatbotscape
Editorial flat-vector illustration for How to Keep Your Chatbot Answering When the AI Model Provider Goes Down
Published 7 min read

How to Keep Your Chatbot Answering When the AI Model Provider Goes Down

Quick answer: Decide which errors should trigger a backup, set a timeout so a hung call becomes an error, retry once, then send the conversation to a second model. Add a circuit breaker so you stop paying the timeout on every request, test the backup on a real set of questions, and write a short honest message for the case where every model fails.

Step 1: Decide which errors trigger a fallback

Write the list down before building. Provider documentation gives a starting point. OpenAI's error-codes guide lists a 500 server error and a 503 "model temporarily overloaded" and advises waiting, then retrying; with a 503 it says to honor a Retry-After header if present. Anthropic lists 500, 504 (timeout) and 529 (overloaded) as server-side conditions.

Treat these as fallback triggers: 500, 503, 504, 529, timeouts, and a rate-limit 429 that is temporary (see handling LLM rate limits). Do not trigger on 400, 401 or 403, which mean your request or key is wrong and will likely fail on the backup too. A spend-cap 429 will not clear by switching models on the same account either. Anthropic also notes that with streaming, an error can arrive after a 200 response has started, so check for errors inside the stream and not only the status code.

Step 2: Set a timeout, then retry once

A hung request is a failure that never announces itself. Set a timeout on every model call, short enough that a customer is still waiting politely. LiteLLM exposes this as request_timeout and the number of retries on the same model group as num_retries.

Keep retries few. Anthropic's SDKs already retry connection errors, rate limits and 5xx errors twice by default, so check your SDK before adding a loop of your own, or attempts multiply.

The time adds up. With an 8-second timeout and one retry, a failed primary costs about 2 × 8 = 16 seconds, plus a 1-second pause between attempts, so about 17 seconds before the backup even starts. If your chat widget needs a first reply sooner, shorten the timeout or skip the retry for 503 and 529. Streaming the answer helps the customer feel progress; see streaming chatbot answers.

Step 3: Add the backup model

Choose the backup for different failure risk, not just a different name. A model from another provider survives a whole-provider outage; a second model from the same provider survives a single overloaded model but not an account-wide problem.

In LiteLLM the ordered list is the fallbacks setting, for example a mapping from your primary model's name to a list of backup names. A gateway is the usual home for it, so application code does not change; see setting up an LLM gateway.

Three checks before you go live:

  1. Prompt compatibility. The same system prompt may behave differently on another model. Test it, and keep a tuned variant if needed; see versioning prompts.
  2. Tools and format. If you rely on function calling or structured outputs, confirm the backup supports them the same way.
  3. Data terms. A backup provider receives customer text. Agree the terms, region and retention first; the data privacy guide lists the questions.

Step 4: Add a circuit breaker

Without one, every request waits out the full timeout on the dead model before falling back. A circuit breaker counts recent failures and, past a threshold, sends traffic straight to the backup for a cooldown period, then tries the primary again. LiteLLM documents allowed_fails and cooldown_time for this.

Pick numbers by reasoning, not copying. If the threshold is too low, a few unlucky requests move all traffic to a pricier backup. If the cooldown is too long, you keep paying for the backup after the primary recovers. Start cautious, watch it for a week, and adjust. Log every trip and every fallback, because a silent fallback can hide a broken primary for days; the monitoring guide covers what to record.

Step 5: Test the failover before you need it

Testing is the step most teams skip. Do it on a test key, not production.

  1. Build a fixed question set. Take 40 to 60 real past questions, including some that call tools and some that should be refused. The regression testing guide shows how.
  2. Force the failure. Point the primary at a bad key or a blocked address, or use your gateway's test mode, so traffic must fall back. Note that LiteLLM's mock fallback flags are ignored on proxy requests in recent versions, per its documentation, so use a real failure there.
  3. Compare answers. Run the set on both models. Check facts, tone, language, refusals and every tool call. Our own pass bar: the backup should be within a few points of the primary on correct answers, and no tool call should break. Set your own bar in advance.
  4. Test long conversations. Switch models halfway through a chat and confirm context carries over.
  5. Time it. Measure how long a customer waits from the first failure to the first backup reply.

Repeat after any change to the prompt, the models or the gateway.

Step 6: Plan for total failure

Sometimes both models fail, or your network does. Decide in advance what a customer sees.

  • Say it plainly. A short message that the assistant is unavailable, with a way forward: leave contact details, view help articles or reach a person. Never show a raw error. See fallback copywriting.
  • Hand off. If staff are available, route to them using the escalation playbook. Warn the team when fallback mode starts, since handoff volume will rise.
  • Scripted mode. For simple tasks such as opening hours or order status, a rule-based path that needs no model can keep working.
  • Hosted platforms. Ask the vendor how it behaves when its model provider is down, and whether any status page or credit applies.

Check the arithmetic

An invented example, not a measurement. A chatbot handles 600 conversations a day. A 90-minute outage falls in a window carrying 15% of daily traffic: 600 × 0.15 = 90 conversations.

  • Without fallback, all 90 fail.
  • With fallback, the backup answers all 90. If the primary resolves 88% without a person and the backup 80%, the gap is 0.08, so 90 × 0.08 = 7.2, about 7 extra conversations go to staff.
  • If the backup costs $0.012 per conversation and the primary $0.008, the extra cost is 90 × $0.004 = $0.36.

About 7 extra handoffs and 36 cents is the price of keeping 90 customers served. Your numbers will differ; the method is to estimate affected traffic, the quality gap and the cost gap.

Common mistakes

  • Falling back on every error. A bad key fails on the backup too.
  • Never testing the backup. The first test is a real outage.
  • No circuit breaker. Every request waits out the timeout.
  • Silent fallback. The primary stays broken, and nobody knows.
  • Forgetting data terms. The backup provider sees customer text.
  • Raw errors on screen.

FAQ

What happens to my chatbot when OpenAI or Anthropic has an outage?

Without a fallback, calls fail and customers see errors or silence. With one, the request moves to a backup model, and a final message covers the case where everything is down.

Should I retry or fall back?

Both, in order: a short retry for brief glitches, then fallback. Keep retries few so customers do not wait long.

Do I need an LLM gateway for fallback?

No, but it keeps the rules out of your application code and handles retries and cooldowns in one place.

Will the backup model sound the same?

Not necessarily. Test tone, language and tool calls before relying on it.

Is a second model from the same provider enough?

It protects against one overloaded model but not a provider-wide outage.

How often should I test failover?

After any change to prompts, models or gateway settings, and on a regular schedule so it does not rot.

Sources

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from the sources above; the steps, the test pass bar and the worked example are our own practice with invented numbers, and we have not benchmarked any provider, SDK or gateway.

Last updated

23 October 2026 — first published.