How to Handle LLM Rate Limits (429 Errors) in a Production Chatbot

Quick answer: Read the error first, because a rate-limit 429 and a billing 429 need opposite responses. For rate limits, honor the provider's wait time, retry with exponential backoff and jitter, and cap the attempts. Then reduce what each call counts, smooth bursts with a queue, add a fallback model or message, and test the limit on purpose before a campaign does it for you.

Step 1: Find your limits and see which one you hit

Open your provider's limits page and write down, for the models you use, the requests per minute, the tokens per minute (input and output separately, where the provider splits them) and any daily or monthly spend cap. OpenAI says limits apply at the organization and project level and vary by model; if you belong to several organizations, confirm the requests use the intended one.

Then log every 429 with its error type, the model, the size of the request and the time. Most teams discover that one limit, often tokens per minute, is the one that binds. The monitoring guide covers where to keep these logs.

Step 2: Tell the two kinds of 429 apart

OpenAI's troubleshooting article distinguishes temporary rate limits from billing or quota errors such as exhausted credits or an organization or project spend limit. Anthropic documents a similar case: reaching a monthly spend cap returns a 429 with no retry-after header until the next month or a tier change.

Build this into your error handler: a rate-limit error is retried; a billing or spend error is not retried and instead raises an alert to a person, since no amount of waiting fixes it. Check the error type or code in the response body, not only the status.

Step 3: Retry with backoff and jitter, and cap it

For rate-limit errors, follow the provider's guidance in this order. If the response carries a Retry-After header, wait at least that long. If not, use exponential backoff with jitter: wait a short time, double it after each failure, and add a random amount so many clients do not retry at the same instant. OpenAI's article recommends capping both retries and total time.

An example schedule (ours, not a standard): waits of about 1, 2, 4 and 8 seconds, each with random jitter, then stop. The base waits sum to 1 + 2 + 4 + 8 = 15 seconds, which is already long for a customer waiting in chat, so decide your own ceiling.

Two cautions from OpenAI's article. Failed requests still count toward per-minute limits, so a tight retry loop makes things worse. And the official SDKs already retry and honor Retry-After, so adding your own loop on top can multiply attempts; check what your SDK does first.

Step 4: Make every call count for less

Because limits often bind on tokens, trimming is the cheapest fix.

1. Set a realistic output cap. OpenAI notes that usage is estimated partly from your maximum output setting, so a large allowance uses capacity even if replies are short. Set it near your typical reply length.
2. Shorten prompts. Remove unneeded instructions and examples, then re-test quality. The token estimation guide shows how to measure what you send.
3. Cache. Anthropic's documentation says that for most of its models, cached input reads do not count toward the input-token limit, so prompt caching can raise throughput there; other providers differ. A semantic cache avoids repeat calls entirely.
4. Send easy questions to a smaller model. Limits are set per model, so routing questions to a cheaper model also spreads load.

Step 5: Smooth bursts, then add a fallback

Bursts fail under the average: both OpenAI and Anthropic say a per-minute limit can be enforced over shorter windows, and Anthropic describes sharp traffic jumps triggering 429s below the normal ceiling.

Queue. Put outgoing model calls through a small queue that releases them at a steady pace under your limit. A queue turns a rejected burst into a short delay. Give each queued customer message a maximum wait, since a customer will not wait indefinitely.

Ramp. For planned spikes such as a broadcast, release the campaign in batches instead of all at once.

Fallback. When the main model is limited and the queue is full, try a second model or provider. An LLM gateway can hold the ordered fallback list so your application code stays simple. Test the fallback's answers, since a different model may behave differently.

Last resort. Show the customer an honest message and an alternative, such as leaving contact details or reaching a person. See fallback copywriting and the escalation playbook. Never show a raw error.

Step 6: Test the limit on purpose

Before a launch or campaign, send a controlled burst on a test key at about double your expected peak and watch what happens: do retries stay under your cap, does the queue drain, does the fallback answer sensibly, do customers see a clean message? Also check your plan: if you want higher limits, providers describe moving up usage tiers, and OpenAI points to a limits settings page and its support route for increases. Tiers change by usage history, so do not assume you can raise the ceiling the day before a campaign.

Check the arithmetic

An invented example, not a measurement. A plan allows 60 requests and 40,000 tokens per minute. Each call sends about 1,800 tokens and allows 500 tokens of output, so it counts as about 1,800 + 500 = 2,300 tokens.

• By requests, 60 calls per minute would fit.
• By tokens, 40,000 ÷ 2,300 is about 17.4, so about 17 calls per minute.
• Raising the output allowance to 1,500 makes each call count 3,300, and the ceiling drops to about 12 (40,000 ÷ 3,300 ≈ 12.1).
• Trimming the prompt by 300 tokens at the 500 output cap gives 1,500 + 500 = 2,000 per call and 40,000 ÷ 2,000 = 20 calls per minute, about 3 more than before.

The token limit, not the request limit, set the ceiling, so trimming and output caps beat asking for more requests.

Common mistakes

• Retrying billing errors. They will not clear by waiting.
• Tight retry loops. Failed calls still count toward limits.
• Double retries. An SDK retry plus your own loop.
• Testing only at low volume. Limits show up under bursts.
• Showing raw errors to customers.

FAQ

How do I fix a 429 error from the OpenAI API?

First read whether it is a rate limit or a billing limit. For a rate limit, wait as the Retry-After header says, retry with backoff and jitter, shrink requests and spread traffic. For billing, add credits or adjust the limit.

What is exponential backoff with jitter?

Waiting longer after each failure, usually doubling, plus a small random amount so many clients do not retry together.

Should I retry forever?

No. Cap the number of retries and the total wait, and then fall back or show a message.

Does a queue slow my chatbot down?

A little, during bursts. Without it, those messages would fail instead, so a short delay is usually better.

Can I just buy a higher limit?

Often by moving to a higher tier or contacting support, but tiers are tied to history, and trimming requests is cheaper and faster.

Does a hosted chatbot platform have rate limits?

The vendor handles the provider's limits but may have plan limits of its own. Ask how it behaves at peak.

Related guides

• Set up an LLM gateway for a support chatbot — fallbacks and retries in one place.
• Route chatbot questions to a cheaper model — spreads load across models.
• Estimate chatbot token usage and cost — measure what each call counts.
• Monitor an LLM chatbot in production — where the 429 logs live.
• LLM rate limit, LLM token and LLM gateway — the glossary entries behind the terms used here.

Sources

• OpenAI Help Center, What are the best practices for managing my rate limits in the API? — help.openai.com/en/articles/6891753, read 21 October 2026: request-size estimation, short-window enforcement, backoff, shorter prompts, tiers.
• OpenAI Help Center, Troubleshooting API rate limits and 429 errors — help.openai.com/en/articles/5955604, read 21 October 2026: rate-limit versus billing errors, Retry-After, jitter, caps, SDK retries, failed requests counting.
• Anthropic, Rate limits — platform.claude.com/docs/en/api/rate-limits, read 21 October 2026: token-bucket refill, acceleration limits, cache-read treatment, spend-limit 429 without retry-after.
• Chatbotscape evaluation methodology. /methodology (continuously updated).

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from the sources above; the steps, the retry schedule and the worked example are our own practice with invented numbers, and we have not benchmarked any provider, SDK or gateway.

Last updated

22 October 2026 — first published.