← All cloud projects
CASE STUDY us-east-1 2026
CLOUD PROJECT · 2026

Four planted faults, found without reading the answer

A CloudFormation stack deliberately broken in four places, debugged live without opening the template. Worked outside-in as a diagnosis log — symptom, hypothesis, evidence, fix — then diffed against the template afterwards to check for both misses and false positives. Four found, zero invented.

CloudFormationALBAuto ScalingVPC routingSecurity Groups

Built, debugged and deleted on 5 August 2026 in us-east-1. Total cost $0.10. The stack no longer exists — the diagnosis log and write-up are in the repo.

The exercise

Deploy a CloudFormation stack that is deliberately broken in four places. Nobody tells you where. Fix it in the live environment, not by correcting the template and redeploying.

The rule I set myself: don’t open the template until it works. It’s the answer key, and reading it converts a diagnostic exercise into a reading-comprehension one. The template is also the only way to check afterwards whether I found everything — and, just as importantly, whether I “fixed” something that was never broken.

Method

Outside-in, as a written log rather than a scavenger hunt. For each fault: symptom → hypothesis → the evidence that confirmed it → the fix. The order follows the request path, because that’s what narrows the space fastest:

  1. Does the load balancer’s DNS resolve at all?
  2. Is there a listener, on the port I think?
  3. Are targets healthy — and what is the health check actually asking for?
  4. Does the security group chain let the balancer reach the instances?
  5. Do the subnets have a route out, and do they span the AZs the balancer uses?
  6. Does the app actually serve on the instance itself — curl localhost?
Operatoroutside-inus-east-1CloudFormationbroken stackVPCInternetgatewayRoute tableAZ aAZ bALB security groupALBlistenerAuto Scaling groupInstance SGEC2web serverInstance SGEC2web serverHTTPforwardforward

Request path, checked outside-in: operator → internet gateway → ALB listener and its security group → instance security group → EC2 web servers in two AZs. Deploy: CloudFormation creates the stack, which was then fixed live rather than redeployed.

requestdeploy / control Hover, tap or Tab to a component to see what it does and why. Swipe sideways for the whole diagram.

Most of these are cheap to check and eliminate whole categories. Step 6 in particular has saved me more time than any other single habit: in another project I spent twenty minutes theorising about client-IP preservation and security groups when the web server had simply never installed.

Result

Four found, four fixed, zero false positives. The diff against the template afterwards confirmed both halves of that — nothing missed, and nothing “corrected” that was actually fine.

In the order the request path exposed them:

  1. An internet-facing load balancer in private subnets. DNS resolved, connections timed out: those subnets had no route to the internet gateway. The fix was moving the balancer to the public subnets. Adding an internet route to the private route table would also have returned 200, and made every private subnet in the VPC public.
  2. A typo in the health-check path, /healthcheck.hmtl. Every check got a 404, so no target could ever be healthy.
  3. The Auto Scaling group was never attached to the target group. Two healthy instances the balancer had never been told about: absent targets, not unhealthy ones.
  4. The instance security group admitted the balancer on port 22, not 80. The right source on the wrong port, which is why it looks correct at a glance.

The second half matters more than it sounds. In an unfamiliar stack there are always things that look wrong and aren’t: an unusual health-check path, a security group that seems too open, a subnet that appears misplaced. Changing them because they look odd is how a debugging session turns into an outage of its own. Every fix here had to be justified by evidence that it was causing the observed symptom, not merely that it looked unusual.

What it’s actually training

Reading someone else’s broken infrastructure with no documentation, no author to ask, and a symptom rather than an error message — which is most of what production debugging is.

The specific faults matter less than the sequence. A health check pointed at a path that returns 404 and a security group missing the balancer’s rule produce the same symptom: unhealthy targets, 503 at the edge. Only the order you check things in separates them quickly.