What is Rate Limiting in AI?
Rate limiting controls how many API requests can be made in a given time period. LLM APIs enforce limits on requests per minute (RPM) and tokens per minute (TPM) to ensure fair usage and prevent abuse.
Why It Matters
Hit rate limits and your application stops working. Understanding limits helps you design resilient systems with proper queuing, retries, and fallbacks. It also prevents runaway costs when agent loops go haywire.
Common Limit Types
- • RPM — Requests per minute (e.g., 60 RPM)
- • TPM — Tokens per minute (e.g., 90,000 TPM)
- • TPD — Tokens per day (total budget)
- • Concurrent — Max simultaneous requests
Handling Rate Limits
Proactive
- • Request queuing
- • Token counting before send
- • Batch requests together
Reactive
- • Exponential backoff
- • Retry with jitter
- • Fallback to smaller model