Handling File Uploads at Scale: Validation, Storage, and Processing
For a marketplace client processing around 40,000 image uploads a day, our first version routed every upload through the API server: validate, resize, write to S3, all in the request-response cycle. It worked at low volume and fell over completely once sellers started bulk-uploading listing photos, with request timeouts cascading into retries that doubled the load.
We split it into two stages. The API now only validates file type, size, and does a quick magic-byte check (not trusting the file extension) before issuing a presigned S3 upload URL directly to the client. This got large files off our application servers entirely — a 15MB photo no longer occupies a request thread while it uploads over a slow mobile connection.
Processing — resizing, thumbnail generation, format conversion to WebP — happens asynchronously via an S3 event triggering a Lambda, writing results back to a separate processed-images bucket. This is also where the six-hour outage happened: a corrupted EXIF header in a user photo threw an unhandled exception in our image library, and because we hadn't set a dead-letter queue, the Lambda kept retrying the same file and blocking the concurrency pool behind it.
We fixed it three ways: wrapped image processing in strict per-file error handling so one bad file can't block others, added an SQS dead-letter queue after three failed attempts, and set a hard concurrency limit so a burst of bad files can't starve the Lambda pool from processing good ones. We also added ClamAV scanning via a Lambda layer for any upload type beyond images, after a different client's document-upload feature briefly became an unintentional file-hosting service for unrelated content.
The remaining constraint is cost: Lambda invocations for image processing run around $180/month at that client's volume, versus roughly $40/month if we ran a dedicated processing box, but the elasticity during traffic spikes (holiday sales pushing uploads to 3x normal) has been worth the premium so far.
