Nightly Terraform drift detector

The problem

“How do you detect configuration drift?” is a stock infrastructure interview question, and the honest answer for most side projects is “I don’t.” Terraform describes what infrastructure should be, but nothing checks that reality still agrees — someone clicks something in the console, a provider upgrade moves a default, a resource gets deleted out of band, and the code quietly becomes fiction.

So: a scheduled terraform plan -detailed-exitcode every night. Exit 0 means reality matches; exit 2 means it doesn’t.

The architecture

A nightly GitHub Actions job assumes a read-only IAM role via OIDC, runs a plan, and reports. Drift opens a GitHub issue and publishes to the existing SNS alerts topic; a clean run closes the issue again.

The role is the interesting part. There are now three OIDC trust paths in this repo, and none can be assumed by a job meant for another:

Role	Trusted on	Can
deploy	`environment:production`	write
PR preview	`pull_request`	write, but only under `previews/*`
plan	`ref:refs/heads/main`	read only

The nightly job runs unattended, on a schedule, with nobody watching. A terraform plan has to read essentially every resource in the account — that is a lot of surface to hand to something that also had write permissions. Separating the roles means a compromised drift workflow can look, and nothing more.

Two more deliberate constraints: it runs plan, never apply — drift is reported, never auto-corrected, because “the code is right and reality is wrong” is a human judgement — and it plans with -lock=false so a nightly job can’t take the state lock and stall a real deploy.

What gets written down

A raw plan is not safe to paste into an issue. It prints attribute values: bucket names, ARNs, account identifiers, tag contents. An issue is permanent and searchable.

So the plan is converted to JSON and reduced to two fields — resource address and action — before anything is written anywhere. Addresses come from the code (aws_s3_bucket.logs[0]), not from the infrastructure, so they carry no account data. There are tests asserting the account id, ARNs and tag values cannot appear in the output, because the whole point of the module is that it’s the boundary between “a plan” and “something published”.

Terraform does mark genuinely secret attributes as sensitive, but “not marked sensitive” is not the same as “fine to publish forever.” The filter is an allow-list of structure, not a deny-list of values.

What it found on its first run

This is the part worth reading. The detector’s first real run reported 21 drifted resources. One was a change I’d made deliberately to test it. The other twenty were bugs — in the detector, and in the repo it was pointing at.

It reported clean while the plan said otherwise. The setup-terraform action installs a wrapper around the terraform binary by default, and that wrapper swallows the exit status. So plan -detailed-exitcode returned 0 while the plan it had just run printed Plan: 0 to add, 21 to change. The issue and SNS steps were gated on that exit code. Every night it would have passed, green, having found real drift and said nothing. That is the worst failure mode a monitoring system has: not “it broke”, but “it lied, quietly, forever.”

Zip packaging is not portable across operating systems. Terraform’s archive_file records each source file’s permission bits, so a Lambda package built on Windows (mode 666) and the identical code built on a Linux runner (644) hash differently. Applies happen from a Windows workstation; the check ran on Ubuntu; all twelve Lambda functions therefore reported as drifted every night. Twenty phantom findings would have buried the one real one — which is how an alert gets muted and stops being read. The fix was to plan on the same OS that applies: a drift detector has to run where the applier runs, or it measures the build host instead of the infrastructure.

Line endings, four separate times. git’s autocrlf rewrites files on checkout, and anything whose bytes get shipped is sensitive to that: a CloudFront function uploaded verbatim, a requirements.txt packaged into a zip, a script imported by the test runner. Each one produced a different symptom — a phantom Terraform diff, a hash mismatch, and my favourite, a SyntaxError pointing at a comment in a completely different file. The repo already had a rule for *.py; it just hadn’t been generalized.

Deployed artifacts depended on shell history. Running the test suite drops __pycache__ and .pytest_cache into the Lambda source directories, and archive_file was packaging them. So what got deployed depended on whether whoever ran terraform apply had run pytest first. The deployed zips literally contained the test files.

None of these were things I suspected. All of them had been true for months.

Cost breakdown

Component	Monthly cost
GitHub Actions (private repo, Windows runner at 2× minutes)	~180 of 2,000 included minutes
Read-only IAM role + STS	$0.00
SNS (reuses the existing alerts topic)	$0.00
AWS API reads from the plan	$0.00
Total	$0.00

What I’d do differently at scale

Pinning the check to a Windows runner works, but it ties CI to whichever OS the applier happens to use — a fragile coupling that’s invisible until someone applies from a different machine. The real fix is to stop applying from a workstation at all: run applies from CI, where the packaging environment is fixed and reproducible, and the drift check runs in that same environment by construction. That needs a broader deploy role and a deliberate approval gate, which is its own project.

Update — that project shipped. Applies now run from CI, so the applier is Linux and this check moved back to ubuntu-latest at half the runner minutes. The write-up, including the broad-role problem and the guardrail I got wrong on the first attempt, is here.