A Config System That Survives Per-Environment Overrides
The incident: during a routine deploy, a config flag meant only for staging — POINT_PAYMENTS_TO_SANDBOX — got merged into a shared config file that both environments loaded, and for about 20 minutes production checkout was silently processing through the sandbox payment processor. No charges failed loudly; they just didn't actually charge anyone, which we only caught because a finance report didn't reconcile the next morning.
The root cause was a single config file with environment blocks (production:, staging:, development:) all living together, edited by hand, with no validation preventing a value from being placed in the wrong block or a key from being typo'd into an unintended section. It worked fine for a long time specifically because nobody made that particular mistake — until someone did.
We rebuilt config loading around a strict base-plus-override model: a base.yaml with every required key and safe defaults, and environment-specific files (production.yaml, staging.yaml) that can only override existing keys, never introduce new ones. A startup-time validation step fails loudly if any environment file references a key not present in the base, or if any required key is missing after merging — catching the class of error we'd hit in production before the app even starts serving traffic.
We also separated genuinely sensitive values (API keys, database credentials) out of these YAML files entirely into the secrets manager, so config files hold structural settings — feature flags, timeout values, third-party endpoint URLs — while credentials never live in a file that could be accidentally shared between environments in the first place. This meant the payment-sandbox-style mistake became structurally harder: sandbox vs. live endpoint is now a secret scoped per environment in Secrets Manager, not a flag in a YAML file anyone could merge incorrectly.
The last piece was making config changes reviewable like code, because they are code: config files live in the same repo, go through the same PR review, and a change to production.yaml specifically requires sign-off from whoever owns that environment. Eighteen months since this change, we haven't had a repeat of an environment-crossing config error, on this client or the others we've applied the same pattern to since.
