Read Replicas and Eventual Consistency: What Breaks If You're Not Careful

We added a read replica to a client's Postgres setup to offload reporting and search queries from the primary, and within a week got a support ticket that looked like a bug: a user updated their profile, refreshed the page, and saw their old data. The write had gone to the primary, but the very next read — served milliseconds later — hit the replica, which hadn't caught up yet. Replication lag was typically under 100ms, but that was enough to produce a visible glitch.

The general pattern we adopted is read-your-writes consistency for anything user-facing immediately after a mutation: route the read that follows a write for the same session back to the primary for a short window (we used 5 seconds), then fall back to the replica for subsequent reads. This isn't a database feature, it's application logic — we track a 'last write timestamp' per session and check it before deciding where to route a read.

Reports and dashboards that aggregate across many users were the safer default case for replica routing, since a few hundred milliseconds of staleness there is invisible to anyone. We routed those unconditionally to the replica and saw an immediate drop in primary load from search and analytics queries that had previously been competing with transactional writes for the same resources.

Replication lag isn't constant, and that's the part that caused a real incident: during a large batch backfill job that generated heavy write volume on the primary, replica lag spiked to over 30 seconds. Any read-after-write logic relying on a fixed 5-second window silently broke during that window, and we started serving stale data on the affected pages. We now monitor replication lag directly and dynamically route to the primary if lag exceeds a threshold, rather than assuming a fixed delay is always sufficient.

The broader lesson was that eventual consistency isn't a database concern you can fully abstract away in the data layer — it has to be a decision made explicitly, endpoint by endpoint, about which reads can tolerate staleness and which can't. We now document that decision directly in the code at the query site rather than leaving it as an implicit assumption someone has to rediscover during an incident.