Incident Response: What We Changed After Our Worst Production Outage
The outage: a routine database migration on a client's e-commerce platform included an index rebuild that we expected to take a few minutes based on staging, where the table was a fraction of production's size. In production, the rebuild took an exclusive lock that held for over four hours, during which every write to that table — meaning every checkout — failed. We had no rollback plan ready because we hadn't seriously considered the migration could go this wrong.
The first hour of the incident was worse than it needed to be simply because of confusion: three engineers investigating independently, no single person clearly in charge of decisions, and about 25 minutes spent figuring out how to even cancel a running Postgres migration safely without corrupting the index. We eventually killed it, but by then the damage in lost checkout revenue and customer trust was done.
The most consequential change afterward was procedural, not technical: every migration touching a table over a defined row-count threshold now requires a dry run against a production-sized snapshot, not just staging, with an explicit rollback command written and tested before the migration is scheduled — not written reactively during an incident. We also now require every high-risk migration to run during a defined low-traffic window with a named engineer on standby specifically for that change, not just general on-call coverage.
We formalized incident command: the first person to notice an incident declares it and either takes incident commander themselves or explicitly hands it off, and that role's only job is coordinating and communicating, not debugging — a separate person drives the technical investigation. This sounds like overhead for a small team, but the second real incident we ran under this structure was resolved in 40 minutes largely because nobody was context-switching between fixing and coordinating.
We also changed what we test. Our staging environment now gets refreshed with a scrubbed, production-scale data snapshot before any schema change touching a large table, specifically because 'it was fast on staging' had been the false signal that let this outage happen. The postmortem itself was blameless and focused on process, but the concrete changes — dry runs at scale, written rollback plans, defined incident command — are what actually stuck, not the apology.
