When a player does something on Players Engine, the request handler does the minimum honest work — validate, write the one record that must exist right now, respond — and publishes an event describing what happened. Everything downstream of that moment (updating derived data, refreshing the graph shadow, sending notifications, recalculating aggregates) happens asynchronously, in a worker service consuming those events from a message queue. This entry is about the discipline that makes that architecture boring instead of haunted.
Why split the work at all
Three reasons, in order of how often they've mattered. Latency: the user's tap responds in the time of one write, not seven. Failure isolation: a notification pipeline having a bad day cannot break posting. Cost: side-effect work batches and retries on its own schedule, on scale-to-zero infrastructure that spins up when there's a backlog and disappears when there isn't. The request path stays sacred; the event path absorbs the mess of the real world.
At-least-once means at-least-twice
Message queues make a promise that sounds reassuring until you parse it: every message will be delivered at least once. Some messages — because of timeouts, crashed consumers, slow acknowledgements, or plain cosmic whimsy — arrive twice or more. The naïve handler that "just processes the message" will eventually double-send an email, double-count a stat, double-create a record. Not often. Just often enough to erode trust in every number the system reports. So the worker's prime directive: every handler must produce the same end state whether it runs once or five times.
Idempotency, concretely
The implementation is less mystical than the word. Derive a deterministic identity for each unit of work — from the event's natural key, not from a random ID minted at processing time — and make the write conditional on that identity not having been processed: create-if-absent instead of create, set instead of increment where possible, and where an increment is genuinely needed, a processed-marker written in the same atomic batch as the increment itself. Redelivery then collapses into a no-op by construction, not by luck. The pattern costs one extra conditional per handler and pays for itself the first week.
Retries lie: re-fetch the truth
The subtler discipline: an event's payload is a photograph of the world at publish time, and by the time a retry runs — possibly minutes later — the world may have moved on. The account in the payload may have been restricted, the content deleted, the permission revoked. Any handler whose action is security- or integrity-relevant re-fetches the current state of the source of truth before acting, and treats the payload as a pointer rather than a fact. This is the asynchronous sibling of the lesson in the auth entry: a snapshot being validly signed, or validly published, doesn't make it currently true.
The topic nobody was listening to
The embarrassing war story. A queue's topology is configuration: topics exist, subscriptions bind consumers to them. During an environment rebuild, a couple of topics came back without their subscriptions — and publishing to a topic with no subscribers is not an error. The publishers reported success. The events sailed into the void. The symptom surfaced far downstream as work mysteriously never happening, and the diagnosis took embarrassingly long precisely because every component was "working". Two lessons got written in permanent marker: topology is code — recreate environments from declaration, not memory — and monitor for the absence of expected work, not just the presence of errors. A pipeline that processes zero events on a day users were active is an alarm, even though nothing failed.
Order is a promise queues don't make
The other property everyone assumes until it bites: delivery order. Two events about the same entity, published seconds apart, can arrive reversed — retries and parallel consumers guarantee it eventually. Handlers here are written to converge regardless: updates carry the source's version or timestamp, and a handler discards work that is older than the state it would overwrite. Design every consumer as if events arrive shuffled, because on a long enough timeline, they do.
The operational seasoning
Acknowledgement deadlines sized to realistic processing time, not optimism, so slow handlers don't trigger spurious redelivery. Exponential backoff with jitter on retries, so a struggling dependency isn't hammered in rhythm. A dead-letter path for messages that fail repeatedly, because one poison message must never wedge a whole subscription. And structured logs that carry the event identity end-to-end, so "what happened to event X" is a query, not an archaeology dig. None of it is clever. Distributed systems don't reward clever; they reward paranoid bookkeeping performed identically every time.