Designing Multi-Region Infrastructure for Lower Latency
A trading-adjacent client with a growing user base in Southeast Asia was seeing round-trip latency of 300-400ms to our single-region API in us-east-1 — mostly speed-of-light distance, not server processing time. For their use case, where users were making time-sensitive decisions on live data, that latency was a genuine product problem, not just a nice-to-fix metric.
We deployed API servers in ap-southeast-1 alongside the existing us-east-1 deployment, with Route 53 latency-based routing sending each user to their nearest region. This dropped round trips for Singapore users to under 60ms. The API layer scaled out easily since it was already stateless behind a load balancer; the database was the real design problem.
We chose a single primary in us-east-1 with a read replica in ap-southeast-1, rather than a multi-primary setup, specifically to avoid conflict resolution complexity. Reads in Singapore hit the local replica; writes route back to the primary in Virginia, meaning writes from Asia still pay the cross-region latency penalty. For this client's usage pattern — heavily read-dominated — that trade-off was the right one, and building true multi-primary would have added complexity we couldn't justify for the write volume involved.
The bugs this introduced were subtler than expected. Replica lag, typically 100-300ms but occasionally spiking to several seconds under load, meant a user could write data via the primary and then immediately read stale data from their local replica. We fixed the cases that mattered — read-your-own-writes after a profile update — by routing a user's own subsequent reads to the primary for a short window after any write from that session, rather than trying to eliminate replica lag itself.
Multi-region also meant rethinking anything that assumed a single source of truth for timing, like rate limiters and idempotency keys backed by Redis, which we moved to a global-ish setup using DynamoDB global tables rather than region-local Redis instances that could disagree with each other. It's more moving parts than a single-region deployment, and we only recommend it once latency is a measured product complaint, not a speculative concern.
