Long-Running Background Jobs Without Blocking the Request Cycle
A logistics client's bulk CSV import feature let users upload up to 10,000 shipment records at once, and the import handler ran synchronously in the request, parsing and validating each row before writing. At small file sizes it worked; at 10,000 rows it routinely took 90-120 seconds, well past our ALB's 60-second idle timeout, so users saw a generic gateway error even when the import had actually succeeded.
We moved import processing to a background job using Sidekiq, with the API endpoint now doing only file validation (correct headers, file size) before enqueueing and returning a job ID immediately. The client polls a status endpoint, and we later added a WebSocket push for progress updates once polling every two seconds started showing up meaningfully in our request logs.
The part that took longer than expected was making imports resumable and partially reportable. Early on, a single malformed row anywhere in the file failed the entire import, which was maddening for a user whose row 9,847 out of 10,000 had a bad date format. We rebuilt it to validate and import row-by-row, collecting failures into a report rather than aborting, so a bad row costs the user one line item instead of the whole file.
For jobs that run longer than a few minutes — we have one client with an overnight inventory reconciliation job that runs 20-40 minutes — we made sure the job itself was idempotent and checkpointed its progress to the database every 500 records. This mattered the first time a deploy happened mid-job: instead of restarting the entire reconciliation, it resumed from the last checkpoint.
The general pattern we now apply by default: anything expected to take more than roughly 2-3 seconds goes into a background job, the API returns immediately with a reference to check status, and the job is built assuming it can be killed and restarted at any point, because eventually it will be.
