Event-driven media pipeline

The problem

The other projects on this platform are read-mostly. This one is the opposite: an interactive, write-path system that a stranger on the internet can drive, which is exactly where event-driven architecture earns its keep — and exactly where the interesting failure modes live. The goal was a real pipeline (not a single do-everything Lambda) that decouples upload from processing, survives a bad input without falling over, and can’t be abused into a bill. All serverless, all inside the free tier at portfolio traffic.

The architecture

The browser never sends bytes to a server. It asks a presign Lambda (behind API Gateway, routed same-origin through the demos CloudFront distribution) for a short-lived, size-pinned presigned S3 PUT, then uploads straight to S3. That decouples the 5 MB payload from the 6 MB Lambda limit entirely.

The upload bucket fires an ObjectCreated event to SQS. A processor Lambda (Python + Pillow) consumes the queue one message at a time: it validates the file is really an image, applies EXIF orientation and strips metadata, writes thumb and medium variants, extracts the dominant color, and publishes a manifest.json. The variants are served back through a new /media/out/* behavior on the demos distribution, and the frontend polls the manifest until they appear.

The queue has a dead-letter queue behind it (maxReceiveCount 3). A malformed file — a .jpg that isn’t an image — fails in the processor, gets retried, and after three attempts lands in the DLQ, where a CloudWatch alarm pages the same SNS topic the observability canary uses.

Key decisions & trade-offs

Presigned direct-to-S3, not bytes-through-Lambda. Routing the file through API Gateway and Lambda would cap uploads at ~4 MB (base64 through the 6 MB synchronous limit) and pay Lambda for every byte. A presigned PUT sends the file straight to S3; Lambda only signs a URL. The cost is one real CSP decision: the demo runs on demos.andruxa.dev, so the browser’s PUT to *.s3.amazonaws.com had to be allowed in the demos distribution’s connect-src. That change is scoped to the relaxed demos CSP only — the main andruxa.dev policy stays strict. The presign endpoint itself is routed same-origin through the distribution’s /api/*, so nothing else needed loosening.

A queue between upload and processing, on purpose. S3 could invoke the processor directly. SQS in the middle buys three things: a retry boundary, back-pressure (the event source mapping caps concurrency), and — the point of the exercise — a dead-letter queue. Poison-message handling isn’t a nice-to-have here; it’s the headline. A file that passes the presign type/size check but isn’t actually a decodable image (Pillow raises UnidentifiedImageError) retries three times and lands in the DLQ with an alarm, instead of silently vanishing or wedging the pipeline. I verified this live: uploading garbage bytes with a .jpg content-type put the message in the DLQ after three failed receives, carrying the exact S3 event that failed.

Pillow bundled, not a public layer. The processor needs a compiled image library. Rather than depend on a community layer ARN outside my control, the deploy zip vendors Pillow’s manylinux / CPython-3.12 wheels via a build script — self-contained, same self-hosting ethos as the fonts on the main site.

Abuse guards before anything is stored. The presign Lambda validates type and size and enforces a per-IP daily quota using a DynamoDB token bucket (an atomic conditional UpdateItem — no read-then-write race), all before handing back an upload URL. Both buckets expire their objects after 24 hours via lifecycle rules, so a stranger can’t fill them and storage trends to zero.

Cost breakdown

What I’d do differently at scale

For real volume the processor moves off a from-scratch Pillow handler to something purpose-built — libvips (via sharp/pyvips) is dramatically faster and lighter on memory for resizing — and the variant set becomes content-negotiated AVIF/WebP with a proper derivative cache rather than eager thumb/medium JPEGs. The presign path grows a virus/content scan step (a quarantine bucket the processor promotes from) before anything is served, and the per-IP DynamoDB bucket graduates to WAF rate-based rules at the edge so abuse is shed before it reaches an API. The DLQ gets a redrive-to-source runbook and an operator dashboard instead of a single alarm. None of that changes the shape here — upload, queue, process, dead-letter — it just hardens each stage.

Links

• Live demo: demos.andruxa.dev/media
• Repo: github.com/andruxap/resume-website