Retries and Circuit Breakers: Surviving Flaky External APIs
The incident that taught us this: a third-party shipping-rate API had a five-minute degradation, and our naive retry logic — three immediate retries on any failure — turned that into a self-inflicted 40-minute outage. Every failed request retried instantly, tripling load on an already-struggling provider, and our own connection pool exhausted itself waiting on slow responses that were mostly going to fail anyway.
We rebuilt retry logic around exponential backoff with jitter: first retry after roughly 200ms, doubling each time up to a 5-second cap, with randomized jitter so that a fleet of our servers retrying the same failure doesn't all hit the provider in the same instant. This alone reduced the load-amplification problem significantly, but it didn't address the case where the provider is down hard rather than just slow.
For that we added a circuit breaker (using the opossum library in Node) in front of every external API call that sits on a critical path. After five consecutive failures within a 30-second window, the circuit opens and fails fast for the next 60 seconds without attempting the call at all, giving the struggling provider room to recover instead of continued load, and giving our own system fast, predictable failures instead of slow ones.
The design decision we spent the most time on was what to do when the circuit is open. For the shipping-rate API, we fall back to a cached rate table that's slightly less accurate but keeps checkout functioning. For a payment provider, there's no safe fallback — an open circuit there just surfaces a clear 'try again shortly' message rather than pretending payment can proceed, since a fabricated success or silent retry there is far worse than a visible failure.
We now apply this pattern to every synchronous call to a system we don't control, with per-provider tuning: a well-behaved internal service between our own teams gets a much shorter timeout and fewer retries than a third-party API known to be occasionally slow. Treating 'external dependency might fail' as the default assumption, not the exception, has prevented at least two more provider outages from becoming ours.
