Building a Job Queue That Survives Worker Crashes
Our first version of a background job system for a client's image processing pipeline used an in-memory queue inside a single Node process. It worked fine until a deploy killed the process mid-job and every queued item — including ones already being processed — just vanished. That's the failure mode that forced us to move to a persisted queue with explicit acknowledgment, backed by Redis with BullMQ for that stack.
The core fix was visibility timeouts: a worker pulls a job and it becomes invisible to other workers for a configured window, say 5 minutes. If the worker crashes before acknowledging completion, the job automatically becomes visible again and another worker picks it up. We tuned the timeout to roughly 3x the p99 processing time for that job type, since too short a timeout causes duplicate processing of jobs that are just slow, not dead.
Duplicate processing is the trade-off you accept with this model, so every job handler in that pipeline had to be idempotent — reprocessing an image that already succeeded needed to be a safe no-op, not a double charge or a corrupted file. We added a processed-at check at the top of each handler as a cheap guard before doing any expensive work.
For jobs that fail repeatedly rather than just timing out, we cap retries at a fixed count with exponential backoff (30s, 2m, 10m, 1h) and then move the job to a dead-letter queue instead of retrying forever. Without that cap, a single malformed input in a batch job spun in an infinite retry loop for six hours before anyone noticed, burning worker capacity that legitimate jobs needed.
The last piece was graceful shutdown: on SIGTERM during a deploy, workers stop pulling new jobs but finish whatever they're holding before exiting, up to a grace period. That single change removed most of the crash-related job loss we were seeing during routine deploys, since deploys — not actual crashes — turned out to be the dominant cause of interrupted jobs.
