Public 'break my site' chaos button

The problem

Project 003 proved this site survives losing its primary region — but I proved it once, by hand, and wrote it up. The honest next question is: do I trust that failover enough to let a stranger trigger it, against production, at any time? Chaos engineering in a slide deck is a claim. Chaos engineering as a public button is a demonstration. So I built the button — and the interesting engineering isn’t breaking the site, it’s the guardrails that make “anyone can break prod” a safe thing to ship.

What happens when you press it

The button POSTs to /api/chaos/drill (same-origin, through the existing CloudFront /api/* behavior — no CSP change). A Lambda then makes the primary origin fail on purpose: it adds an explicit Deny on s3:GetObject for CloudFront to the site bucket’s policy. The primary bucket now returns 403 to the edge, and because 403 is in the 003 origin group’s failover criteria, CloudFront serves the us-west-2 replica instead — in the same request, no DNS change, no user-visible error. Healing removes the Deny.

The page then polls GET /api/chaos every couple of seconds and streams the state: primary down → failing over to the replica → dead-man heal counting down → healed. You watch the site take a bullet and keep standing.

Visitor presses the button → API Gateway → drill Lambda. The Lambda takes a DynamoDB lock (1/hour), schedules an INDEPENDENT one-shot heal, then flips the site bucket policy to Deny → primary 403 → CloudFront origin group serves the us-west-2 replica. The one-shot EventBridge Scheduler removes the Deny ~5 min later, even if the Lambda died.

The guardrails are the project

Letting the public flip production unhealthy is only sane because of four controls, each doing a specific job:

Dead-man auto-heal, created before the break. The drill must always heal, even if the Lambda crashes the instant after it flips the primary. So the Lambda creates an independent one-shot EventBridge Scheduler — a separate identity, a separate execution — before it adds the Deny. If the Lambda dies mid-drill, the one-shot still fires in ~5 minutes and removes the Deny. ActionAfterCompletion=DELETE makes each one self-clean after firing. The heal path does not depend on the happy path.

One drill per hour, globally. A DynamoDB conditional write takes a lock keyed on an expires_at timestamp; a second press inside the hour fails the condition and returns 423 Locked with the remaining cooldown, which the page renders as a disabled button and a countdown. A botnet can’t keep the region flapping.

It can’t run without a replica to catch it. The whole stack is gated in Terraform on multi_region_enabled. If the DR replica isn’t live, the button cannot be deployed — because flipping the primary with nowhere to fail over to would be a real outage, not a drill.

A GET can never break the site. The trigger is POST only; the Lambda routes GET to the read-only status endpoint and POST to the drill. A crawler, a link-prefetch, or a preview screenshotter hitting the URL gets status JSON, not an outage.

The live drill

Here is a real press of the public button, measured end to end on production:

	Result
Trigger	`POST /api/chaos/drill` → `202 failing-over`
Primary during the window	`us-east-1` returning `403` to CloudFront (`ChaosDrillDeny` on the bucket policy)
`andruxa.dev` during the outage	`200` on every path — the `/chaos/` page and the homepage both kept serving
Served from	the us-west-2 replica, at ~49 ms
Auto-heal	fired on its own 5m19s later — no manual action — and removed the `Deny`
Second press during the window	`423 Locked`, ~59 min cooldown shown in the UI
After the drill	bucket policy back to a single `Allow`; one-shot schedule self-deleted; `terraform plan` clean

The headline is the same as 003’s, now on demand and initiated by anyone: no visitor ever saw an error. The measurable cost of a regional failure here isn’t downtime — it’s a few hundred milliseconds of extra latency while the edge serves a farther replica. That’s the trade-off origin-group failover buys, and a public button is the most honest way I can think of to keep proving it.

Key decisions & trade-offs

Add a Deny, don’t delete the policy. The 003 drill deleted the whole bucket policy by hand. For an automated, public drill that rewrites a production policy on every press, that’s too blunt — a bug could drop the legitimate Allow permanently. Instead the Lambda makes a surgical edit: it adds exactly one statement (Sid: ChaosDrillDeny) and, on heal, removes exactly that statement. The Terraform-managed Allow is never touched, so the worst case is a stale Deny that the dead-man (or the next terraform apply) cleans up.

Heal is idempotent and doesn’t hold the lock. Healing when already healthy is a no-op, so the dead-man firing after a manual heal does nothing harmful. And healing (which happens at ~5 min) deliberately does not release the hourly lock — the cooldown outlives the failover window, so the “1/hour” guarantee holds regardless of how fast a drill heals.

Single DynamoDB table for lock + history. The cooldown lock and the drill history live in one on-demand table (partitioned LOCK vs DRILL, TTL-swept). The status endpoint reads the history for the page’s timeline and recent-drills log; the drill path takes the lock. No second data store for what is a tiny amount of state.

Cost breakdown

Component	Monthly cost
Standby region + health mechanics	$0 new — already paid by 003
Drill Lambda + DynamoDB lock/history (on-demand)	~$0.01 at the 1/hour cap
EventBridge Scheduler (dead-man one-shots)	well under the free tier
Total	≈ $0.01 incremental

What I’d do differently at scale

A real chaos program wouldn’t stop at one blast type. I’d turn the button into a menu of failure injections — kill a Lambda, throttle a table, blackhole an AZ, expire a cert — each with its own blast-radius limit and dead-man restore, and I’d run them on a schedule as game days rather than only on demand, tracking RTO/RPO against targets over time. The bigger jump is AWS Fault Injection Service, which does this with typed experiments, stop conditions, and IAM-scoped blast radius instead of my hand-rolled bucket-policy flip. The principle that survives at any scale is the one this button demonstrates: a failover you don’t continuously break is a failover you don’t actually have.