Designing Idempotent APIs for Safe Retries
A client's Flutter app was creating duplicate orders under bad cell coverage — the request would succeed server-side, the response would time out client-side, and the app would retry the same POST. The server had no way to know it was the same intent twice. We added an Idempotency-Key header, generated client-side as a UUID at the moment the user tapped 'place order,' and required it on every mutating endpoint that touched money or inventory.
The server stores the key alongside the request hash and the resulting response for 24 hours. A retry with the same key returns the cached response instead of re-executing the handler. A retry with the same key but a different request body — which would indicate a client bug rather than a network retry — gets rejected with a 422 rather than silently doing something unexpected.
The tricky part isn't the happy path, it's the in-flight case: what happens if a retry arrives while the first request is still processing. We handle this with a unique constraint on (idempotency_key, tenant_id) at the database level and let the second request's insert fail fast, then poll briefly for the first request's result rather than racing it.
Not every endpoint needs this. We only apply idempotency keys to endpoints with side effects that are expensive or hard to reverse — payments, order creation, inventory decrements. Read endpoints and naturally idempotent operations like PUT-based updates don't need the machinery, and adding it everywhere just adds storage and latency for no benefit.
The other lesson: idempotency has to be designed into the endpoint, not bolted on later. Retrofitting it onto an existing checkout flow meant auditing every side effect — emails sent, webhooks fired, inventory adjusted — to make sure a cached response replay didn't also replay those. We ended up wrapping the side-effecting calls in a transactional outbox so retries only replay the HTTP response, not the downstream actions.
