Building AI Features That Degrade Gracefully When the Model Is Down
A provider outage lasting about 40 minutes took a client's platform down almost entirely, even though the AI feature involved — product search refinement — was meant to be a nice-to-have enhancement layered on top of standard search. The problem was that our error handling for a failed AI call was to retry, then retry again, then surface a generic error page, rather than falling back to the standard search that had worked fine before the AI feature existed.
We rebuilt the failure path around a simple principle: every AI-enhanced feature needs a non-AI fallback that existed and worked before the AI feature was added, and that fallback needs to activate automatically and fast, not after several retries have already burned through the user's patience. For search refinement, that meant falling back to the previous keyword-based ranking within 2 seconds of a detected failure, not 40 minutes of retries and error pages.
We set explicit circuit-breaker thresholds rather than retrying indefinitely — after three consecutive failures within a 30-second window, the circuit opens and all traffic routes to the fallback for a cooldown period, with periodic health-check calls to detect recovery before flipping traffic back. This kept us from hammering an already-struggling provider with retries, which in a prior incident had actually made our own recovery slower once the provider came back up.
For features with no reasonable non-AI equivalent — a chat assistant, for instance — graceful degradation meant something different: a clear, honest status message instead of a hung or generic-error UI, and disabling the entry point rather than letting users start an interaction that would fail partway through. We'd rather a feature look temporarily unavailable than let someone invest two minutes in a conversation that then fails.
We now require a documented fallback plan as part of design review for any new AI feature, the same way we'd require an error state for a database call. The provider outage was the wake-up call, but the actual fix — treating AI calls as a dependency that fails like any other external service, not as an infallible layer — should have been there from the start.
