Background Job Retries: Exponential Backoff and Dead-Letter Queues
Our early retry logic on a client's job processing system retried failed jobs immediately, up to five times, with no delay between attempts. During an outage of a third-party shipping rate API, every job that depended on it failed and retried instantly, generating a burst of retry traffic against an already-struggling service — the retries themselves were making the outage worse and taking longer to resolve than if we'd simply backed off.
We switched to exponential backoff with jitter: delays roughly doubling each attempt (30s, 1m, 2m, 4m, 8m) with a random offset of up to 20% added to each delay to avoid every failed job in a batch retrying at exactly the same moment and re-creating the same thundering herd problem in a smaller, delayed form. That jitter detail mattered more than we expected — without it, jobs that failed together tended to retry together, recreating bursts at each backoff interval.
Not every failure deserves the same number of retries. We classify failures into transient (network timeout, 503 from a downstream service) which get the full backoff sequence, and permanent (validation error, 400 response, resource not found) which fail immediately without retrying, since retrying a request that will never succeed just delays the failure notification and wastes worker time.
Jobs that exhaust their retry budget move to a dead-letter queue rather than disappearing, which was the gap in our original system — jobs that failed five times just logged an error and were gone, with no easy way to see what had failed or reprocess it once the underlying issue (like that shipping API outage) was resolved. The dead-letter queue keeps the full job payload and failure history, and we built a small internal tool to inspect and manually replay dead-lettered jobs once a root cause is fixed.
We also added alerting on dead-letter queue depth rather than relying on someone noticing it manually, since a slow leak of failed jobs is easy to miss day to day but adds up to real customer impact over a week. A threshold alert (more than 20 jobs dead-lettered in an hour) has caught issues — like a schema change on a downstream API that broke a specific job type — well before customers reported anything.
