Applying Terraform from CI, not from my laptop

The problem

Everything else on this platform deploys through CI with no long-lived credentials. The site, the demo assets, the Lambda code, the short links — all of it flows through GitHub Actions and OIDC. Except the infrastructure itself, which I applied by hand, from a Windows workstation.

That has two costs, and the visible one is the smaller one.

The visible cost: the drift detector was measuring my laptop. Terraform’s archive_file records each source file’s permission bits in the zip, so a Lambda package built on Windows (mode 666) and byte-identical code built on Linux (644) hash differently. Applies came from Windows, so the nightly check had to run on windows-latest too — otherwise every Lambda reported as drifted, every night, burying anything real. That pin worked, and it was a symptom fix: it tied CI to whichever operating system the applier happened to be using.

The real cost: an apply could ship code that wasn’t in git. Nothing about running terraform apply locally checks that your working tree is committed, or that your branch isn’t stale. This is not hypothetical here. Earlier in this platform’s life an apply ran from a stale branch and silently reverted a round of production security hardening — OIDC trust widened, an S3 permission re-added, prevent_destroy guards dropped. I caught it and corrected it, but “I caught it” is not a control.

CI can only apply what is committed. That’s the whole thesis: the fix isn’t discipline, it’s removing the opportunity.

The architecture

One workflow, workflow_dispatch only, main only. No push trigger, no schedule, no pull request. It plans, summarizes, and — only if you asked it to — applies.

dispatch (mode: plan | apply, confirm: "apply")
  → refuse unless ref is main
  → check the typed confirmation          ← before credentials exist
  → assume the apply role via OIDC
  → terraform plan -out=ci.tfplan
  → render a value-free summary
  → terraform apply ci.tfplan             ← the SAVED plan, never a re-plan
  → SNS notify · scrub every artifact

Two details carry most of the safety.

It applies the saved plan file, never a fresh one. What was summarized is exactly what runs. If state moved between the plan and the apply, Terraform refuses — and that refusal is the feature.

Plan and apply live in the same job, so the plan file never becomes a downloadable artifact. A raw plan prints attribute values: bucket names, ARNs, account identifiers. The sibling drift detector already refuses to publish those, and splitting this into plan → approve → apply jobs would have forced me to upload one.

The role, and why it can’t be small

There are now four OIDC trust paths, each trusted on a different token subject, so a job built for one physically cannot assume another:

The environment: infrastructure line in the workflow is load-bearing rather than decorative: declaring it is what puts that string in the OIDC token subject, and that string is the only thing the role trusts. Remove the environment and the workflow can’t assume the role. No other workflow declares it, so none of them can reach it either.

An apply role for this stack needs write access across about fifteen services plus IAM. There is no least-privilege policy that both works and is small, and I’d rather say that than pretend otherwise. An enumerated allow-list would be long, brittle, and would fail the moment a new resource type appeared — and a half-finished apply is worse than a broad role.

So the containment is layered, not scoped. The role is PowerUserAccess plus the IAM it genuinely needs, and then explicit Deny statements block the escalation paths:

• No self-modification. It cannot grant itself permissions, widen its own trust policy, or delete its own guardrails.
• No rewriting any OIDC role’s trust policy. Those four trust conditions are the separation in the table above; anything that could rewrite them could grant itself the others.
• No IAM users, no access keys, no login profiles. This project’s rule has always been “deploys are OIDC-only, never a static credential.” This makes that a property of the account rather than a convention in a document — a compromised role can’t mint something to persist with after the hour-long session ends.
• No attaching AdministratorAccess. Otherwise the shortest escalation is: create a new role, attach admin, assume it, and every deny above stops mattering.
• No touching CloudTrail or account-wide posture. A separate project grades those; an apply that could quietly switch them off would turn that grade into a self-report.

Explicit Deny beats every Allow in IAM’s evaluation, including PowerUserAccess, so these hold however the allow side grows later.

There’s a deliberate consequence: changing this role, or any trust policy, now requires a workstation apply. The one thing CI must not be able to rewrite is the definition of what CI may do.

The guardrail that looked present and protected nothing

I originally wrote the deny list by hand-typing the four role ARNs. The PR-preview role is github-actions-previews-…; I typed github-actions-preview-…, singular.

IAM accepts a Deny on an ARN that doesn’t exist. It’s valid, it applies cleanly, and it silently protects nothing. terraform validate, tfsec, and the plan were all green, and the guardrail was visible in the diff — while the real preview role’s trust policy stayed fully writable by CI, because the broad allow granted UpdateAssumeRolePolicy across every role in the account.

I found it because terraform output printed the actual ARN next to my constructed one. The fix is to never hand-write an ARN for a resource Terraform already manages: reference it. A wrong reference fails to compile; a wrong string fails silently, and only ever in the direction of less security.

The general lesson is about how you verify a Deny. Proving it needs simulate-principal-policy against the correct resource type — asking about CreateAccessKey while passing a role ARN returns implicitDeny, which looks reassuring and only means the action and resource never matched, so your deny was never evaluated at all. explicitDeny is the pass.

The gate

The textbook answer is a GitHub Environment with required reviewers. On a free private repo that’s a paid feature — and a solo maintainer approving his own change isn’t a second pair of eyes, it’s a speed bump you paid for.

So the gate is: manual dispatch, main only, with the word apply typed by hand into a form field. The confirmation is checked before credentials are assumed, so a mistyped one never reaches AWS at all. Two independent mechanisms have to agree the ref is main — the environment’s branch policy and an explicit check in the workflow.

mode defaults to plan, which stops after printing the summary. Applying is the deliberate second action. That split turned out to matter more than I expected.

The transition, and what it taught me

The first apply from Linux had to rewrite every Lambda package hash, so I ran mode=plan first and read it. Good thing:

Plan: 0 to add, 20 to change, 0 to destroy.

Thirteen Lambda functions — expected, the permission-bit churn. But also seven IAM policies I hadn’t predicted, all of them scheduler roles. Nothing about an inline IAM policy should depend on the operating system that ran Terraform.

They turned out to be structurally identical: one statement each, granting lambda:InvokeFunction on an unqualified function ARN — the kind that doesn’t change when code changes. And each mapped one-to-one onto one of the thirteen changing Lambdas.

That’s the tell. Each policy is built by a data source that references its Lambda’s ARN. When the Lambda has a pending change, Terraform defers the data source to apply time, so the policy plans against (known after apply) and is counted as a change even though the result will be identical.

That was a hypothesis, so I wrote down how it would be falsified — apply, then re-plan; a clean second plan means they were no-ops, seven remaining changes means I was wrong — and then ran it:

Apply complete! Resources: 0 added, 13 changed, 0 destroyed.

Thirteen, not twenty. The seven weren’t even written. The follow-up plan came back No changes, every live endpoint still returned 200, and the drift detector moved to ubuntu-latest and reported clean.

The habit I’d keep from this: a plan count is an upper bound, not a measurement, and an unexplained entry in it is worth ten minutes before you apply rather than an hour after.

The polarity flipped

There’s a genuinely confusing side effect worth writing down where someone will find it. Now that Linux is the applier, a terraform plan from my Windows workstation reports those thirteen Lambdas as drifted. The workstation is wrong; the infrastructure is fine. Local plans are a rough check now, and the note lives in the drift workflow’s own comments — the place you’d actually be looking when confused.

I also managed to demonstrate the original problem while fixing it: I applied Phase A to live AWS from a branch that wasn’t merged yet. Harmless, because I closed the gap in the same session, and precisely the ordering this project exists to make impossible. Once the workflow shipped, it stopped being possible.

Cost breakdown

What I’d do differently at scale

The approval gate is the weak point, and it’s honest to say so. Typing a word into a form is a gate against accident, not against a determined attacker who already has repo write access. With a team, this becomes a required-reviewer rule on the environment; the workflow needs no change, only the rule.

The role is broader than I’d accept with more than one applier. With several stacks, I’d split it per stack so an apply to one can’t reach another, and put a permissions boundary on the roles Terraform creates. I considered a boundary here and rejected it: a boundary only binds roles that carry it, and attaching one to fifty existing execution roles is a large, risky change to buy a property the explicit denies already give.

And the packaging problem is still papered over. One fixed applier environment makes the Windows/Linux hash difference irrelevant rather than fixed. The actual fix is building Lambda zips deterministically — fixed modes, fixed timestamps — so the hash depends only on the code. Then no environment is special, and the whole class of question disappears instead of being pinned to one answer.

Links

• The workflow: infra-apply.yml
• The role and its guardrails: apply-from-ci.tf
• The project this one grew out of: nightly drift detector
• Repo: github.com/andruxap/resume-website