PlayersEngine runs its side-effects through Pub/Sub: a user action hits an API service, the service does the minimum synchronous work, publishes an event, and a background worker picks it up to do everything else — updating counters, writing notifications, syncing the social graph. I've written before about why that split exists. This post is about the unglamorous piece that makes it survivable for one person: the dead-letter queue.
What a DLQ actually buys you
When a Pub/Sub push delivery fails — the worker returns an error, times out, or is simply down — Pub/Sub retries with backoff. Retries solve transient problems: a cold start, a brief database hiccup, a deploy in progress. But some messages don't fail transiently. A malformed payload, a bug in one handler path, a downstream record that doesn't exist — these fail every time. Without a dead-letter policy, a poison message like that retries forever, burning delivery attempts and, worse, sitting in front of healthy messages while the subscription churns.
A dead-letter queue is the escape valve: after N failed delivery attempts, Pub/Sub moves the message to a separate topic instead of retrying eternally. The rule I settled on is boring and absolute: every push subscription gets a DLQ, no exceptions. Not because every subscription will need one, but because the moment you need one and don't have it, the message is already gone — acked into the void or endlessly looping. A DLQ you never use costs nearly nothing. A missing DLQ costs you data, and you don't get to choose which data.
The three-legged stool
DLQs only work as part of a trio, and each leg covers the others' weaknesses:
Retries handle transient failure. They're the first responder, and most failures end here.
Idempotent handlers make retries safe. Pub/Sub is at-least-once delivery — the same message can and will arrive twice, especially around retry storms. Every handler in the worker is written so that processing a message twice produces the same result as once: deterministic IDs for created records, conditional writes, "already done? exit cleanly." I re-learned the importance of this during a staging incident where a database was unavailable for a while and a large retry backlog replayed all at once. Because the handlers were idempotent, the replay was a non-event. If they hadn't been, the recovery would have been a second incident — double-created records, double-counted stats.
The DLQ catches what retries can't fix. It converts "infinite retry loop" into "message parked, system healthy, human decides later."
Remove any leg and the other two get worse: without idempotency you fear retries; without retries every blip dead-letters; without a DLQ your poison messages never leave the stage.
The honest confession
Here's the part that ops-mature companies won't tell you and I will: an unmonitored DLQ is just a slower way to lose data. Moving a message to a dead-letter topic doesn't fix anything — it politely postpones the loss. If nobody ever looks, the only difference between dead-lettering a message and dropping it is the storage bill and your self-image.
I know this because I've been on the wrong side of it. In an infrastructure review of my own staging environment, I found dead-letter queues that had been faithfully collecting messages with no alerting on any of them. The mechanism worked perfectly; the process around it didn't exist. Every one of those parked messages represented a side-effect some user action was still waiting on — silently.
So the real requirement is DLQ plus two process rules. First, visibility: some signal a human actually receives when a dead-letter topic goes from empty to non-empty. It doesn't need to be fancy — a metric threshold that pings a channel I already read is enough. The bar is "I find out within hours," not "I find out during the next audit." Second, a replay path: dead-lettered messages must be re-processable once the underlying bug is fixed. This is where idempotency pays double — replaying a batch that includes some already-half-processed work is only safe because handlers tolerate duplicates.
What a one-person ops rotation really looks like
I'm the on-call rotation. All of it. That constraint shapes the design more than any scalability target, and it's why I lean on this machinery so hard. The system has to degrade in ways that wait for me.
Synchronous failures can't wait — a user is staring at them — so the synchronous path stays minimal. Everything that can be deferred is deferred into Pub/Sub, precisely because deferred work has a safety net under it: retries absorb the failures that fix themselves, the DLQ holds the ones that don't, and nothing requires me to be awake when it happens. A poison message at 3 a.m. parks itself; I read about it over coffee, fix the handler, replay the queue. The alternative architecture — where side-effects happen inline and their failures are the user's problem — assumes an ops team I don't have.
The takeaway
If you're building event-driven systems solo, the priority order is: idempotent handlers first, DLQs on every subscription second, alerting on DLQ depth third — and you're not done until all three exist. The first two are one-time engineering costs. The third is the one everyone skips, and it's the difference between a safety net and a hammock slowly filling with everything you dropped.