Multi-region serverless DR

Try it → Every response names the region that served it. Writes are deleted automatically after 24 hours.

# read — tells you which region answered
curl https://api.lab.andruxa.dev/health

# write in one region, read it back from the other
curl -X POST https://api.lab.andruxa.dev/ \
  -H 'Content-Type: application/json' \
  -d '{"ItemId":"hello","Data":"written from anywhere"}'

The architecture

Primary path (top): Route 53 primary record → us-east-1 API Gateway → Lambda → DynamoDB replica. Failover path (bottom): the primary health check fails twice → secondary record → the identical us-west-2 stack. One global table, replicated between the regions.

requestasync / eventfailure / failover Hover, tap or Tab to a component to see what it does and why. Swipe sideways for the whole diagram.

One table, not two that sync. A global table is a single logical table with a replica in each region; DynamoDB owns the replication. It’s the only piece of the stack that isn’t duplicated, and it’s the reason data survives losing a region at all.

What was measured

The primary was broken for real, not simulated: Lambda reserved concurrency set to 0, so every invocation throttles and the API returns 503. An actual compute failure, and fully reversible.

	Break	Restore
Users switched region	90 s	120 s
Alarm changed state	270 s	300 s

Two separate things worth separating.

Users moved in 90 seconds. Route 53’s health check runs at a 30-second interval with a failure threshold of 2, so ~60 seconds is the detection floor before it even begins serving the secondary, and resolver caching adds the rest. That’s the real recovery time, and it matches the 60–80 seconds measured on the original build.

The alarm arrived three minutes after that — and that’s correct, not a bug. Route 53 acts on its own health checks directly; the CloudWatch metric is a downstream publication with its own lag, plus two 60-second evaluation periods on top. The alarm’s job is to tell me to look, not to perform the failover. By the time it fires, users have been served by the other region for minutes.

Confusing those two numbers is how people end up quoting an RTO that’s three times worse than reality — or, worse, wiring an alarm into an automated response and discovering it acts long after the thing it’s reacting to has resolved itself.

Failback works too, which is a separate claim needing its own test: plenty of setups fail over correctly and never come home.

The trap that would have broken it

Never point a health check at the record it controls.

If the primary’s health check resolved api.lab.andruxa.dev, then: the primary fails → DNS moves to the secondary → the primary’s check now resolves to the secondary → reports healthy → traffic flaps back to a region that’s still broken.

So each check targets its own region’s execute-api hostname directly. That also matters because a health check sends the FQDN it’s configured with as the Host header, and the custom domain would reject an unexpected one.

A DR setup that can’t fail back is worse than one that can’t fail over, because it looks fine.

Details worth knowing

Alias records pointing at API Gateway can’t use EvaluateTargetHealth, so each failover record needs an explicit health check attached.

Regional API Gateway needs its certificate in its own region — this is not the CloudFront us-east-1 rule. Two certs, one per region, both validating against the same DNS name.

servedBy in every response is what made this debuggable. Having each region stamp its identity into the payload turned “did failover happen?” from a guess into a string comparison. The same trick appears in the deployment measurements and the routing project.

Running it in public

Everything here is pay-per-request, so idle genuinely costs nothing. The only fixed charge is two Route 53 health checks at $0.50 each — and those are exactly the resources that survive every other teardown, which is why the teardown script deletes them first.

Guarded like the other live demo: 5 req/s throttle on both regions’ APIs, and a 24-hour TTL on every write so the table can’t grow without bound. The TTL attribute is epoch seconds; milliseconds would mean the year 56000 and nothing would ever be deleted. DynamoDB replicates the expiry deletions to the other region like any other write.

How this compares