Backups Are Not Disaster Recovery Until You've Tested the Restore
We took over infrastructure for a client whose previous team had set up nightly RDS snapshots and considered disaster recovery solved. Snapshots were running, green checkmarks every night for two years. Nobody had ever restored one. When we ran a scheduled DR drill, the first restore attempt failed because the snapshot's storage tier had been changed at some point and the restore process silently pointed at an inaccessible bucket path.
After fixing that, the second attempt succeeded but took just over nine hours for a 400GB database, far longer than the four-hour recovery time objective the client's contracts implied to their own customers. Nobody had measured this before because nobody had tried. We hadn't set an explicit RTO or RPO target for this client before the drill either — we'd inherited 'backups exist' as the entire DR posture.
We restructured around explicit numbers: a 1-hour RPO (recovery point objective, meaning at most an hour of data loss is acceptable) achieved via continuous WAL archiving to S3 rather than nightly snapshots alone, and a 45-minute RTO achieved by keeping a warm standby replica in a second region that can be promoted rather than restored from scratch. The standby costs roughly $340/month extra — a real cost the client was willing to pay once they saw the nine-hour number.
We now run a DR drill quarterly, not annually, because the first drill also surfaced things unrelated to the database itself: DNS TTLs set too high to fail over quickly, an application config hardcoded to a specific region's endpoint, and an on-call runbook that referenced a restore procedure two infrastructure changes out of date. Each drill since has found at least one smaller issue, which is exactly the point.
The rule we hold clients to now: a backup you haven't restored is a hypothesis, not a plan. We schedule the first real restore test before we consider a backup strategy done, not as a nice-to-have afterward, because the failure modes — permission changes, storage tier drift, expired credentials — tend to be invisible until the moment you actually need them not to be.
