← All cloud projects
CASE STUDY us-east-1 2026
CLOUD PROJECT · 2026

Proving a private service is actually private

Three isolated VPCs sharing one internal service over PrivateLink. Showing the consumers can reach it is the easy half — the claim worth testing is that they reach it while the provider has no internet gateway, no peering, and an address they cannot touch. Seven assertions, four of them negative. Also: the private instance couldn't install its own web server.

VPCPrivateLinkNLBVPC Endpoint ServiceGateway EndpointsSSM

Built, measured and torn down on 7 August 2026 in us-east-1. Total cost $0.23. The infrastructure no longer exists — the code, the validation script and the write-up are in the repo.

The setup

Three VPCs with no connection between them. Shared Services runs an internal app on a private-subnet instance with no public IP, behind an internal Network Load Balancer, published as a VPC Endpoint Service. Payments and Analytics each consume it through an Interface Endpoint.

No peering. No transit gateway. No NAT.

us-east-1Payments VPCInstanceconsumerInterfaceendpointAnalytics VPCInstanceconsumerInterfaceendpointShared Services VPCno IGW · no NAT · no peering · no TGWEndpointserviceNLBinternalPrivate subnetEC2 appno public IPSSM endpoints3 interfaceS3 gatewayendpointS3Systems ManagerPrivateLinkPrivateLinkno route:80dnf

Service path: consumer instance → interface endpoint → PrivateLink → endpoint service → internal NLB → private app. Negative (red): no route from a consumer to the provider's network. Bootstrap and admin (bottom right): an S3 gateway endpoint for dnf, SSM interface endpoints for Session Manager.

requestfailure / failover Hover, tap or Tab to a component to see what it does and why. Swipe sideways for the whole diagram.

The validation

The course this came from is missing its final-testing lesson, so I wrote the validation from scratch — and that turned out to be the most useful part, because it forced the question of what actually needs proving.

Demonstrating that a consumer can reach the provider is trivial and proves almost nothing. PrivateLink’s claim is stronger: the consumer reaches it without the traffic touching the internet and without the two networks being connected at all. So four of the seven assertions are negative:

1. both consumers reach Shared Services through PrivateLink        PASS
2. the provider has no public IP                                   PASS
3. the provider's subnet has no internet route, and its VPC has
   no internet gateway at all                                      PASS
4. zero peering connections, zero transit gateways                 PASS
5. a consumer CANNOT reach the provider's private IP directly      PASS
6. the endpoint DNS resolves to an ENI inside the consumer's
   own VPC — 10.10.1.217, not the provider's address               PASS

Tests 4 and 5 are what give test 1 any meaning. Anyone can show two machines talking; the claim here is that they talk while genuinely disconnected.

The finding

PrivateLink is one-directional exposure, not connectivity.

Peering joins two networks and both sides gain routes into the other. PrivateLink gives the consumer an ENI representing one service — it never learns a route to the provider’s CIDR and cannot reach anything else there. Test 5 proves it: the provider’s own IP is unreachable from a consumer that is happily using its service.

That asymmetry is the security property, and it’s why CIDR overlap between the two VPCs wouldn’t matter.

ExposesCIDR overlapCost shape
PrivateLinkone service, one directionfineper endpoint-hour + per GB
Peeringthe whole VPC, both waysbreaksdata transfer only
Transit Gatewaywhole VPCs via a hubbreaksattachment-hour + per GB

Peering for a few VPCs that trust each other completely. Transit Gateway when many VPCs need general routing. PrivateLink when you want to publish a service to consumers you don’t want inside your network — a vendor, another business unit, a regulated boundary.

What broke

The first validation run failed test 1 while every negative passed. The load balancer target was unhealthy, and my first instinct was NLB client-IP preservation and security groups — with instance targets the app sees the consumer’s address rather than the balancer’s, so a group scoped to the provider’s CIDR would break correct traffic.

That was wrong. Checking the instance first:

$ systemctl is-active httpd
inactive
$ tail /var/log/cloud-init-output.log
Failed to restart httpd.service: Unit httpd.service not found.

The web server had never installed. dnf install httpd failed because the instance sits in a private subnet with no NAT and no internet gateway — by design — and package repositories are on the internet. The architecture that makes the exercise’s point also prevents the exercise from building.

The fix is an S3 gateway endpoint: Amazon Linux 2023’s repositories are served from S3 in-region, and gateway endpoints are free, unlike interface endpoints at ~$0.01/hr per AZ. dnf install then worked with still no NAT and no IGW anywhere.

The generalisable lesson: a private instance needs a deliberate path for its own bootstrap — NAT gateway (~$32/mo), gateway endpoint (free, S3 and DynamoDB only), interface endpoint, or bake an AMI. And the debugging lesson is cheaper still: check whether the thing is running before theorising about why traffic to it fails.

The cost trap

Five interface endpoints existed here — two for the consumers, three for SSM so the isolated instance could be administered at all. Each bills ~$0.01/hr per AZ, and they appear in no EC2 console view, only under VPC → Endpoints. That’s roughly $36/month of invisible spend if a teardown misses them, which is why teardown deletes them first and counts them last.

Administering that instance is itself worth noting: SSM reached a machine with no internet path via ssm, ssmmessages and ec2messages interface endpoints — PrivateLink used on AWS’s own APIs, which is how you manage genuinely isolated infrastructure.