Zero-Downtime Deploys: Blue-Green, Canary, and Rolling Strategies
Rolling deploys — replacing instances a few at a time behind a load balancer, waiting for health checks before moving to the next batch — are our default for most client projects because they need no extra infrastructure beyond what a standard container orchestrator already provides. The trade-off is that a bad deploy is visible to a subset of real users for however long it takes the rollout to reach the point of failure, which for a fast-moving rollout might only be a minute or two, but is still real exposure.
Blue-green deploys, where a full second environment is stood up and traffic is cut over all at once, gave us the cleanest rollback story — if something's wrong post-cutover, flipping traffic back to the previous environment is instantaneous rather than requiring a re-deploy. We used this for a client where the database migration accompanying most releases made partial rollout risky; either the whole new environment was correct against the new schema, or none of it was live yet.
The real cost of blue-green is running two full production-sized environments simultaneously during the cutover window, which for a client with an expensive GPU-backed inference service made blue-green meaningfully pricier per deploy than we were comfortable doing multiple times a day. For that project we used canary instead — route 5% of traffic to the new version, watch error rates and latency for 15 minutes, then ramp to 25%, 50%, 100% if metrics hold.
Canary deploys require the most supporting infrastructure of the three — traffic splitting at the load balancer or service mesh level, automated metric comparison between the canary and baseline versions, and a defined rollback trigger, ideally automatic rather than requiring someone to be watching a dashboard at 2am. Building that automated rollback (error rate exceeding baseline by more than 2x for 3 consecutive minutes triggers an automatic revert) was what made canary trustworthy enough to actually rely on rather than just a slower rolling deploy.
For database migrations specifically, we treat them as a separate concern from the deploy strategy entirely — using the expand-and-contract pattern so the schema is compatible with both the old and new application code during any rollout window, regardless of whether that rollout is rolling, blue-green, or canary. Coupling a migration to a specific deploy strategy is what causes rollbacks to become genuinely hard, since rolling back application code against an already-migrated schema is a common way deploys go from routine to incidents.
