Production

What is Rate Limiting in AI?

Rate limiting controls how many API requests can be made in a given time period. LLM APIs enforce limits on requests per minute (RPM) and tokens per minute (TPM) to ensure fair usage and prevent abuse.

Why It Matters

Hit rate limits and your application stops working. Understanding limits helps you design resilient systems with proper queuing, retries, and fallbacks. It also prevents runaway costs when agent loops go haywire.

Common Limit Types

  • • RPM — Requests per minute (e.g., 60 RPM)
  • • TPM — Tokens per minute (e.g., 90,000 TPM)
  • • TPD — Tokens per day (total budget)
  • • Concurrent — Max simultaneous requests

Handling Rate Limits

Proactive

  • • Request queuing
  • • Token counting before send
  • • Batch requests together

Reactive

  • • Exponential backoff
  • • Retry with jitter
  • • Fallback to smaller model

Related Terms

← Back to Glossary