The problem
Everything else on this platform lives in us-east-1. That’s a single-region bet, and the honest question for anyone claiming to do reliability work is: what actually happens if that region — or just my one bucket in it — goes away? “It’d probably be fine” isn’t an answer. This project makes the answer demonstrable: a second region holds a live copy, the edge fails over automatically, and I proved it by breaking production on purpose and measuring what happened.
The architecture
The site bucket in us-east-1 replicates continuously to a replica bucket in us-west-2 via S3 Cross-Region Replication. The same CloudFront distribution now wraps both S3 origins in an origin group: the primary is the us-east-1 bucket, the secondary is the replica, and CloudFront fails over to the replica when the primary returns 403, 404, or a 5xx.
The key property is where the failover happens. This is origin-level failover inside CloudFront, not DNS-level failover through Route 53. DNS failover reacts to a health check and then waits out a record’s TTL before clients pick up the new answer — seconds to minutes. Origin-group failover happens within a single request: CloudFront tries the primary, gets an error, and immediately retries the secondary before responding. There’s no window where the user gets an error and no TTL to wait out.
Breaking us-east-1 on purpose
A DR setup you haven’t tested isn’t a DR setup. So I ran a drill against production.
To simulate the primary becoming unreachable, I deleted the site bucket’s policy — the statement that lets CloudFront’s Origin Access Control read it. With that gone, the primary bucket returns 403 AccessDenied to CloudFront for every object. I cleared the edge cache with an invalidation so requests would actually reach the origin, then watched.
What happened, measured end to end:
| Result | |
|---|---|
| Primary state during the window | 403 to CloudFront (bucket policy deleted, confirmed NoSuchBucketPolicy) |
andruxa.dev during the outage | 200 on every path — /, /about/, /work/cloud/, /work/software/ |
| Served from | the us-west-2 replica |
| Added latency | ~0.42s vs ~0.2s baseline (≈ 200 ms) |
| User-facing downtime | none |
| Primary-down window | ~68 seconds (14:07:59Z → 14:09:07Z), start to full restore |
Then I restored the bucket policy with a plan-reviewed terraform apply and re-invalidated; latency dropped back to the ~0.2s baseline as the edge returned to the primary, and terraform plan came back clean — no drift from the exercise.
The headline: RTO was effectively sub-second. There was no moment where a visitor got an error. The measurable cost of failover wasn’t downtime — it was ~200 ms of extra per-request latency (CloudFront trying the primary first, then reaching a farther, colder replica). That is exactly the trade-off origin-group failover makes versus DNS failover, and it’s the right one for a site that must not blink.
Key decisions & trade-offs
Origin-group failover, not Route 53 DNS failover. DNS failover is the more familiar pattern, but it’s slower (health-check interval + record TTL) and it fails the whole distribution over. Origin groups fail over per-request, inside the edge, with no client-visible DNS change. The trade-off: 403 has to be in the failover criteria for the bucket-policy drill (and for a real “primary can’t be read” failure) to trigger — which means a genuine 404 also probes the replica before the custom 404 page renders. Real pages are unaffected; only missing ones pay a small penalty. Worth it.
No Route 53 health check — and 006 is why. The plan called for one, but the primary origin is a private OAC bucket: there’s nothing anonymous to health-check, and the account-level S3 Public Access Block from the security scorecard project deliberately forbids the public probe object such a check would need. That’s two of my own controls interacting — the security posture constrains the DR design — so monitoring rides on the existing synthetic canary and the CloudFront 5xxErrorRate alarm instead. If failover ever failed, users would get 5xx and that alarm would page; during this drill it stayed quiet because failover worked.
us-west-2, not another continent. A replica in Europe would tell a slightly better “region-independent” story, but us-west-2 is the classic DR pair with us-east-1: genuinely separate blast radius, but low enough replication cost and failover latency to keep the whole thing in the pennies. CRR only replicates objects created after it’s enabled, so I did a one-time aws s3 sync to backfill the current site into the replica; every deploy since replicates automatically.
Cost breakdown
| Component | Monthly cost |
|---|---|
| Replica S3 storage (~2 MB site) | fractional cents |
| Cross-region replication transfer | ~$0 at this size |
| CloudFront origin group | $0 (no extra charge) |
| Total | ≈ pennies/mo |
What I’d do differently at scale
For a larger system the replica stops being a cold copy of a static bucket and becomes a warm standby of the whole stack — the API, the data stores, the Lambdas — replicated per-service (DynamoDB global tables, Aurora global database), with the DR drill promoted from “delete a bucket policy” to a scheduled game day that fails a real region and measures RTO/RPO against a target. I’d add Route 53 Application Recovery Controller for coordinated, safe regional evacuation rather than relying only on edge origin failover, and I’d wire the drill into CI so failover is re-proven on a schedule instead of once. The pattern here — replicate, fail over automatically, and test it by breaking things — is the part that scales unchanged.
Links
- The live site (this is the artifact): andruxa.dev
- Repo: github.com/andruxap/resume-website