place — a live collaborative pixel canvas

The problem

Almost every cloud portfolio is request/response: a page, an API, a form. Real-time — a persistent connection, state shared live across strangers — is the thing that’s hard and that nobody ships. So I built the honest test of it: a shared pixel canvas. Anyone can place a pixel, and everyone connected sees it the instant it lands. It forces every real-time question into the open — connection management, broadcast fan-out, abuse control, and how to keep a live socket from wrecking a security posture I care about.

The architecture

An API Gateway WebSocket API holds the persistent connections. One Lambda handles the whole protocol, routed by requestContext.routeKey:

• $connect registers the connection (with a hard concurrency cap) and stamps its source IP.
• $disconnect drops it.
• draw validates a pixel, takes a per-IP cooldown, writes the pixel, and broadcasts it to every open connection.

All state lives in one DynamoDB table, partitioned by kind: connection registry, canvas rows, and per-IP cooldown markers. A scheduled Lambda flattens the canvas to a single JSON snapshot in S3, so a cold page load is one GET instead of replaying history. The page itself is static, on its own CloudFront distribution.

Key decisions & trade-offs

One Lambda, routed by routeKey. Three separate functions for connect/disconnect/draw is the tutorial shape, but it triples the cold starts and scatters the shared state logic. One function routed on requestContext.routeKey keeps the whole protocol — registry, cooldown, broadcast — in one readable place.

Atomic single-pixel writes, no read-modify-write. The naive canvas store is one big blob you read, mutate one byte of, and write back — which loses pixels under concurrent draws. Instead each canvas row is a DynamoDB list and a draw is SET px[i] = colour: a single atomic update of one index, no read, no lost-update race. The row is lazily created blank on its first pixel.

Per-IP cooldown as a conditional put. The r/place rule — one pixel every few seconds — is a DynamoDB conditional write on a per-IP marker (attribute_not_exists OR expired). It’s the same primitive I used for the chaos-drill lock: the database enforces the rate limit atomically, so there’s no window to race. Abuse test: 256 rapid draws from one IP landed exactly 2 pixels over 5 seconds.

A hard connection cap. Broadcast fan-out is O(connections), and connections are the cost driver. $connect refuses past a ceiling, which bounds both the per-draw fan-out work and the bill against a connection flood.

Its own subdomain, so the main CSP never bends. A browser can’t open a WebSocket unless the page’s CSP connect-src allows the wss:// endpoint. Rather than punch that hole in andruxa.dev’s strict CSP, the canvas lives on place.andruxa.dev — its own CloudFront distribution, its own dedicated ACM cert, and its own headers policy that is identical to the main site’s except for the one wss:// addition. The portfolio’s CSP stays pristine; the relaxation is quarantined to the one page that needs it. Giving it a dedicated cert also means the shared main/demos certificate is never re-issued.

Reaping dead connections — and a bug that taught me the edge case. Broadcasting to a client that has silently dropped throws GoneException; those get reaped from the registry. I extended reaping to BadRequestException (an invalid/stale connection id) too — and promptly broke connections, because the peer-count broadcast fired during a client’s own $connect posts to the still-connecting socket, that post fails, and the new client got reaped a millisecond after joining. The fix: exclude the connecting socket from its own connect broadcast. (An earlier bug in the same stack: an auto_deploy WebSocket stage that deployed before its routes existed, so every handshake 403’d with zero Lambda invocations until the stage depended on the routes.)

Cost breakdown

What I’d do differently at scale

At real r/place scale the single broadcast Lambda becomes the bottleneck — fanning out to hundreds of thousands of sockets from one function doesn’t hold. I’d move fan-out off the write path: writes go to DynamoDB, a DynamoDB Stream drives the broadcast, and connections are sharded across workers so no single invocation posts to every socket. The canvas itself would move from row-items to a binary representation with server-side batching of pixel deltas, and the cooldown would graduate from per-IP to real identity to survive NAT and mobile networks. The shape here — persistent connections, atomic writes, bounded fan-out, abuse control at the edge — is the part that carries; only the fan-out mechanism has to change.

Links

• Draw on it (this is the artifact): place.andruxa.dev
• Repo: github.com/andruxap/resume-website