OpenAI's rate-limit help articles and Anthropic's rate-limits documentation were read on 21 October 2026. Limits, tiers and error names change often, so check your provider's current limits page.

LLM rate limit· LLM operations

An LLM rate limit is a cap a model provider places on how much your account can send in a window of time, usually counted in requests per minute and tokens per minute. Exceed it and the API rejects the call, typically with an HTTP 429 error, until capacity frees up.

By Chatbotscape Editorial· Methodology· Published 22 October 2026· Updated 22 October 2026

LLM Rate Limit — Why a Busy Chatbot Can Fail While Nothing Is Broken

Quick answer: A rate limit is the provider's ceiling on your usage per unit of time. When a chatbot built on a model API sends more than the ceiling allows, the provider refuses the extra calls with a 429 error. The chatbot is not broken and the model is not down; it is being told to slow down. The fix is a mix of smaller requests, patient retries, spreading traffic out and, if needed, a higher limit.

What is being counted

Providers count more than one thing at once, and you can hit any one of them first.

Requests per minute (RPM): how many calls you make, regardless of size.
Tokens per minute (TPM): how much text you send and receive. Anthropic's documentation splits this into input tokens per minute and output tokens per minute; see LLM token for the unit.
Other caps: some providers also set daily limits or monthly spend limits, tracked separately.

OpenAI's help article adds a detail that surprises people: usage is estimated from the size of the request, meaning the prompt plus the maximum output you allow, so a large output allowance can use up capacity even if the reply turns out short. Anthropic's documentation says its output-token count is based on tokens actually generated, so providers differ.

Why bursts fail under the average

Limits quoted per minute are often enforced over shorter windows. Both OpenAI's and Anthropic's pages say so: a 60-per-minute limit can behave like one request per second, and Anthropic describes capacity that refills continuously (a token bucket) rather than resetting on the minute. A chatbot that gets twenty messages in the same second after an email campaign can be rejected even though its hourly average is tiny. Anthropic also documents "acceleration" limits, where a sharp jump in traffic can trigger 429s below your normal ceiling.

Two different 429 errors

OpenAI's troubleshooting article separates the cases, and the difference decides what you do.

Kind	Cause	Retrying helps?
Rate limit	Too many requests or tokens in a short window	Yes, after a wait
Billing or quota	Credits used up, or an organization or project spend limit reached	No, fix the balance or limit first

Anthropic documents a similar case: when a monthly spend cap is reached, requests return a 429 with no retry-after header, and access returns on the first of the next month or after a tier increase. A retry loop that treats this as a temporary error will spin uselessly, so read the error type, not just the status code.

What helps, and what does not

Helps: waiting at least as long as a Retry-After header says; exponential backoff with jitter when there is no header; shorter prompts; a realistic output cap; spreading requests out; and a higher usage tier. Anthropic's documentation notes that for most of its models, cached input reads are not counted toward the input-token limit, so prompt caching can raise effective throughput there. Check your provider's rule, because it varies.

Does not help: resending failed requests in a tight loop. OpenAI's article warns that failed requests still count toward per-minute limits. Stacking your own retry loop on an SDK that already retries can also multiply attempts.

Where a gateway fits

An LLM gateway can queue calls, retry them and move traffic to a second model when the first is limited, so the application does not handle each provider's quirks. That is a design choice with its own cost; it does not raise your limit, it manages what happens at it. The idea of picking a model per request is covered in LLM routing.

Worked example (invented numbers)

Suppose a plan allows 60 requests and 40,000 tokens per minute. Each chatbot call sends about 1,800 tokens and allows up to 500 tokens of output, so each counts as roughly 1,800 + 500 = 2,300 tokens.

By request count, 60 calls per minute would fit.
By tokens, 40,000 ÷ 2,300 is about 17.4, so about 17 calls per minute.
The token limit binds first, at roughly 17 calls, well below 60.
If the output cap were raised to 1,500, each call would count as 3,300 and the ceiling would fall to about 12 calls per minute (40,000 ÷ 3,300 ≈ 12.1).

The lesson is that the limit you hit is often the token one, and trimming prompts and output caps buys more headroom than most people expect.

FAQ

What is an LLM rate limit?

A cap on how many requests and tokens a model provider accepts from your account in a period, usually a minute.

What does a 429 error mean?

Too many requests, in the HTTP standard sense. For LLM APIs it usually means a rate limit, but it can also mean a billing or spend limit, which a retry will not fix.

What is the difference between RPM and TPM?

RPM counts calls. TPM counts the tokens inside them. A few very large requests can exhaust TPM while RPM looks fine.

Does a higher tier remove the limit?

No. It raises it. Providers place accounts in tiers by usage or spend history, and higher tiers have higher limits, but there is always a ceiling.

Do retries count against my limit?

Per OpenAI's troubleshooting article, failed requests still count toward per-minute limits, which is why backoff matters.

Can I avoid rate limits by using a chatbot platform?

The platform absorbs the provider's limits, but it may have its own plan limits or slow down under load. Ask the vendor how it behaves at peak.

LLM token — the unit the TPM limit counts.
LLM gateway — a layer that queues, retries and falls back.
Prompt caching — can reduce counted input on some providers.
Semantic caching — avoids repeat calls altogether.
LLM routing — sends each request to a suitable model.
Chatbot API — the interface these limits apply to.

Sources

OpenAI Help Center, What are the best practices for managing my rate limits in the API? — help.openai.com/en/articles/6891753, read 21 October 2026. Source of request-size estimation, per-second enforcement, backoff, shorter prompts and tier advice.
OpenAI Help Center, Troubleshooting API rate limits and 429 errors — help.openai.com/en/articles/5955604, read 21 October 2026. Source of the rate-limit versus billing distinction, Retry-After, jitter, and failed requests counting toward limits.
Anthropic, Rate limits — platform.claude.com/docs/en/api/rate-limits, read 21 October 2026. Source of RPM, input and output tokens per minute, the token-bucket behavior, acceleration limits, cache-read treatment and the spend-limit 429.
Chatbotscape evaluation methodology. /methodology (continuously updated).