Rate Limiting and Queueing Strategies for LLM-Backed Endpoints
An LLM-backed feature on a client's platform went down during a marketing push that drove a sudden traffic spike, and the postmortem showed the model API itself never actually failed — our own request handling did. We'd applied the same rate-limiting middleware used for the rest of the API, which was tuned for calls that complete in tens of milliseconds, not LLM calls that routinely took two to six seconds and held a connection open the whole time.
The fix started with separating LLM-backed endpoints onto their own request queue with its own concurrency limits, distinct from the general API rate limiter. We capped concurrent in-flight LLM calls at a number tuned to the provider's actual rate limit rather than our server's connection capacity, since the provider's limit was the real bottleneck and we'd been hitting it invisibly, retrying, and compounding the backlog.
We added a priority queue rather than strict FIFO, because not all requests were equal — a user actively waiting on a chat response needed to jump ahead of a background batch summarization job. Requests get a priority tag at submission time, and the queue worker drains high-priority requests first, with a fairness mechanism so batch jobs don't starve indefinitely under sustained interactive load.
For users, we replaced the previous behavior of a request either completing or timing out with an explicit queue-position response when the system was under load, so the client-side UI could show 'processing, position 4 in queue' instead of a spinner that looked identical whether the wait would be two seconds or twenty. That single UX change eliminated most of the 'is this broken' support tickets we'd been getting during load spikes.
We also added backpressure at the queue itself — once the queue depth passed a threshold, new low-priority requests get a fast, honest 'try again shortly' response instead of being accepted and left to time out later. Accepting more work than the system could actually process in reasonable time was the root cause of the original outage, not the LLM provider being slow.
