Health Checks That Actually Catch Real Failures
The health check on a client's checkout service was a classic shallow one: a /health endpoint that returned 200 OK as long as the process was running and could respond to HTTP at all. It never actually queried the database, called the payment provider, or exercised anything close to what checkout actually does. Predictably, it stayed green through an incident where the database connection pool had been exhausted and every real checkout request was failing.
The load balancer kept routing traffic to instances reporting healthy for twelve minutes while customers saw failures, and our on-call only found out from a support ticket, not from monitoring, since every dashboard tracking 'instance health' looked fine. The gap between 'the process is alive' and 'the service can do its job' was the entire problem.
We rebuilt health checks in two tiers. Liveness checks stayed shallow on purpose — just 'is the process responsive' — because that's what determines whether an orchestrator should restart a container, and a slow database shouldn't cause a healthy process to get killed and restarted, which would make an already-degraded situation worse. Readiness checks became deep: verifying an actual (cheap, read-only) database query succeeds within 200ms, that the connection pool has available connections, and that a lightweight ping to the payment provider's status endpoint succeeds.
Readiness failures now pull an instance out of the load balancer rotation without killing the process, so it stops receiving new traffic but can recover on its own once the database pool frees up, rather than being restarted and losing whatever in-flight work it had. This distinction — liveness for 'should this process exist,' readiness for 'should this process receive traffic' — is one we now build into every client project from the start rather than retrofitting after an incident.
We also stopped trusting the load balancer's view alone and added a synthetic transaction check running every two minutes from outside the infrastructure — an actual scripted checkout against a test product, hitting every dependency a real customer would. It's caught two subsequent issues (a DNS misconfiguration for the payment provider, and a certificate that expired a day early) that instance-level health checks would never have seen, because the failure was in something between the service and the outside world, not in the service itself.
