How to handle LLM API rate limits in production

Why Rate Limits Break Production Systems — and What's Actually at Stake#

When you integrate a large language model API into a production application, rate limits are not an edge case. They are a certainty. OpenAI, Anthropic, Google, and every major LLM provider enforce rate limits at multiple levels — requests per minute (RPM), tokens per minute (TPM), and sometimes daily caps. The moment your application gains real users, you will hit these ceilings.

The consequences of mishandling rate limits go beyond a degraded user experience. In B2B contexts, where clients depend on your platform for automated workflows, document processing, or AI-assisted tooling, a single unhandled 429 Too Many Requests error can cascade into failed jobs, corrupted state, and lost revenue. Engineers who treat rate limits as a deployment afterthought spend their on-call rotations doing damage control.

The Anatomy of a Rate Limit Response#

Most LLM providers return a 429 HTTP status code when a limit is exceeded. The response body typically includes:

• Error type: e.g., rate_limit_exceeded or quota_exceeded
• Retry-After header: seconds to wait before retrying (not always present)
• Error message: human-readable explanation of which limit was hit

Here is an example response from the OpenAI API:

{
  "error": {
    "message": "Rate limit reached for gpt-4o in organization org-xyz on tokens per min. Limit: 30000, Used: 29800, Requested: 1500.",
    "type": "tokens",
    "code": "rate_limit_exceeded"
  }
}

Understanding what triggered the limit — RPM versus TPM — determines your mitigation strategy. Hitting token limits requires different handling than hitting request limits.

The Business Cost of Unhandled Errors#

In a multi-tenant SaaS environment, a single burst from one client can exhaust your organization-level quota and take down service for every other tenant. This is the silent risk that most teams underestimate. Rate limits are shared across your entire API key unless you implement per-client key isolation or usage-based routing. For B2B platforms processing high volumes of AI requests, this is not a hypothetical — it is a structural vulnerability.

Beyond availability, there is the cost of wasted compute. If your application retries immediately without backoff, you burn through your remaining quota in milliseconds, making the problem worse. Naive retry logic is often the primary reason a temporary rate limit becomes a sustained outage.

---

Core Strategies for Handling Rate Limits Reliably#

Robust rate limit handling requires layering multiple strategies. No single technique is sufficient on its own. The goal is to build a resilient request pipeline that degrades gracefully under pressure rather than failing hard.

1. Exponential Backoff with Jitter#

Exponential backoff is the foundational retry pattern for any API that enforces rate limits. When a 429 is received, the client waits before retrying, doubling the wait time with each subsequent failure. Jitter — randomized variance in the wait time — prevents the thundering herd problem, where multiple clients retry simultaneously and immediately re-trigger the limit.

Here is a Python implementation:

import time
import random
import openai

def call_with_backoff(prompt: str, max_retries: int = 6) -> str:
    base_delay = 1.0
    for attempt in range(max_retries):
        try:
            response = openai.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": prompt}]
            )
            return response.choices[0].message.content
        except openai.RateLimitError:
            if attempt == max_retries - 1:
                raise
            delay = base_delay * (2 ** attempt) + random.uniform(0, 1)
            time.sleep(delay)

This pattern caps at roughly 63 seconds of total wait time across six attempts. Adjust max_retries and base_delay based on your SLA requirements.

2. Token Counting Before Dispatch#

Request-level rate limits are straightforward to track, but token limits require proactive management. Sending a request without estimating its token cost is like driving without checking the fuel gauge. Use a tokenizer library — tiktoken for OpenAI models — to count tokens before dispatching.

import tiktoken

def estimate_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

With token counts available, you can implement pre-flight checks that reject or queue requests exceeding a safe threshold before they ever reach the API.

3. Request Queuing with Rate-Aware Throttling#

For high-throughput applications, a client-side queue with a rate-aware dispatcher is essential. Instead of sending requests as fast as they arrive, the dispatcher tracks current usage against your known limits and holds requests when approaching the ceiling.

Libraries like ratelimit for Python or bottleneck for Node.js provide token-bucket or leaky-bucket implementations that enforce a maximum request rate client-side:

const Bottleneck = require('bottleneck');

const limiter = new Bottleneck({
  maxConcurrent: 5,
  minTime: 200 // minimum 200ms between requests = max 5 RPM at this setting
});

const throttledCall = limiter.wrap(callLLMAPI);

Combined with backoff on actual 429 responses, this dual-layer approach — proactive throttling plus reactive retry — handles the vast majority of real-world rate limit scenarios.

4. Caching Repeated Requests#

A significant portion of LLM requests in production are semantically identical or near-identical. Caching responses for common prompts reduces API calls, cuts costs, and eliminates rate limit exposure for repeated queries. Use a semantic cache like GPTCache or a simple Redis-backed exact-match cache for deterministic prompts:

import hashlib
import redis
import json

cache = redis.Redis(host='localhost', port=6379, db=0)

def cached_llm_call(prompt: str, ttl: int = 3600) -> str:
    key = hashlib.sha256(prompt.encode()).hexdigest()
    cached = cache.get(key)
    if cached:
        return json.loads(cached)
    result = call_with_backoff(prompt)
    cache.setex(key, ttl, json.dumps(result))
    return result

For B2B applications where users frequently run similar queries — report generation, document classification, template-based content — cache hit rates of 20–40% are realistic and meaningfully reduce API pressure.

---

Production Architecture Patterns for Scale#

Once your application moves beyond a single service making direct API calls, you need architectural patterns that enforce rate limit handling consistently across your entire infrastructure. Ad-hoc implementations scattered across microservices create inconsistent behavior and make quota management nearly impossible.

Centralized API Gateway for LLM Calls#

Route all LLM requests through a single internal gateway service. This gateway owns the API keys, enforces global rate limits, applies retry logic, and exposes usage metrics. Individual services call the gateway rather than the LLM provider directly.

Benefits of this pattern:

• Centralized quota management: One place to monitor and control usage across all services and tenants
• Key rotation without service changes: Rotate API keys in the gateway without touching downstream services
• Consistent retry and backoff behavior: No risk of different services implementing conflicting retry strategies
• Cost attribution: Tag requests by service, tenant, or feature to understand where quota is being consumed

Open-source projects like LiteLLM can serve as the foundation for this gateway, providing a unified interface across multiple LLM providers with built-in load balancing and fallback routing.

Multi-Provider Fallback Routing#

Relying on a single LLM provider creates a single point of failure. When OpenAI's API experiences degraded performance — which happens — your application goes down with it. Multi-provider routing uses a primary provider under normal conditions and fails over to an alternative when rate limits or errors are detected.

def resilient_llm_call(prompt: str) -> str:
    providers = [
        lambda: call_openai(prompt),
        lambda: call_anthropic(prompt),
        lambda: call_google_gemini(prompt)
    ]
    for provider in providers:
        try:
            return provider()
        except (RateLimitError, APIError):
            continue
    raise Exception("All LLM providers exhausted")

This pattern requires normalizing prompts and responses across providers, which adds implementation overhead but delivers meaningfully higher availability for production B2B workloads.

Async Job Queues for Non-Real-Time Workloads#

Not every LLM call needs to be synchronous. Batch processing, document summarization, content generation pipelines, and background enrichment tasks can all be moved to async job queues. This decouples request ingestion from API dispatch, allowing the queue worker to process at a controlled rate regardless of upstream burst traffic.

A Celery-based worker with rate limiting:

from celery import Celery
from celery.utils.rate_limits import rate

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task(rate_limit='30/m', max_retries=5, default_retry_delay=10)
def process_document(document_id: str):
    doc = fetch_document(document_id)
    result = call_with_backoff(build_prompt(doc))
    store_result(document_id, result)

The rate_limit='30/m' decorator enforces a maximum of 30 task executions per minute at the worker level, providing hard client-side rate enforcement before requests ever reach the API.

Observability and Alerting#

Rate limit handling is invisible without proper instrumentation. Track the following metrics in your monitoring stack:

• Rate limit hit rate: 429 responses as a percentage of total requests
• Retry count distribution: How many retries are needed before success
• Queue depth: Backlog size for async workloads
• Token utilization: Tokens used versus your per-minute limit, tracked as a time series
• Provider fallback activations: How often fallback providers are invoked

Set alerts when the 429 rate exceeds 5% of requests or when queue depth grows beyond a defined threshold. These signals indicate that your current rate limit tier is insufficient for your traffic volume and that a tier upgrade or architectural change is needed before the situation becomes an outage.

Expose these metrics via Prometheus and visualize them in Grafana, or use a managed observability platform like Datadog or New Relic with custom LLM-specific dashboards. The investment in observability pays for itself the first time you catch a rate limit issue before it escalates.