This is a postmortem about an outage where nothing was broken. No bug, no bad deploy, no quota blown. The database had simply — by design, with my consent, documented in the pricing page I'd skimmed — gone to sleep.
The symptoms
PlayersEngine keeps social relationships in Neo4j (the "who follows whom" shape that document databases are bad at) alongside Firestore for documents. One morning in staging, everything touching the social graph started failing at once. Likes wouldn't register. The background worker that processes side-effect events from Pub/Sub was erroring on every delivery, and its retries were piling up.
The error in the logs was ENOTFOUND — a DNS resolution
failure on the database hostname. And this is where I lost most of
the incident time, because of what ENOTFOUND
implies. DNS failure reads like an infrastructure problem:
something wrong with the container's networking, a misconfigured
connection string from a recent change, maybe a stale driver
holding dead connections. Those were my first three theories, and I
worked through them in order. I diffed recent commits for anything
touching database config. I restarted services to rule out poisoned
connection pools. I stared at the driver's retry settings.
All application-layer theories. All wrong.
The cause
The staging environment runs on a managed Neo4j Aura instance on a low tier — appropriate for staging, where traffic is a trickle of test activity. What I had mentally filed away and then completely forgotten: low-tier Aura instances auto-pause after a period of inactivity. A paused instance doesn't refuse connections or return a helpful "I am paused" error. Its endpoint effectively stops resolving. From the application's point of view, the database doesn't error — it vanishes.
Staging had been quiet for a stretch. The instance hit its inactivity threshold and paused, exactly as its tier promises. Every service that touched the graph then failed with a DNS error that looked, for all the world, like a networking bug.
The fix took under a minute: open the console, click resume, wait for the instance to come up. Then the actual cleanup began — draining the retry backlog that had accumulated while the graph was gone, which is its own small lesson in why idempotent event handlers are non-negotiable. Replaying a burst of held-back deliveries is only safe when handling the same event twice is harmless.
Lesson one: check managed-infra state first
My debugging instinct — built from years where the database was a process on a machine I controlled — is that databases are up, and errors mean my code is wrong. With managed services, that prior is outdated. A managed instance has a lifecycle of its own: it pauses, resizes, migrates, gets maintenance windows. Its state is a first-class suspect.
The rule I wrote down afterwards: when a dependency starts failing wholesale, check the provider's console before reading a single line of application code. Thirty seconds in a dashboard beats an hour of well-intentioned code archaeology. Wholesale failure — every operation, every service, all at once — is itself a fingerprint. Bugs are usually selective; infrastructure is indiscriminate.
Lesson two: dev tiers have dev-tier behaviours
The auto-pause wasn't a flaw. It's the reason the staging tier is affordable, and I'd re-choose it today. The flaw was that the behaviour lived in the provider's documentation instead of in my operational memory. Free and low-cost tiers of managed services routinely have semantics their production siblings don't: instances that sleep, quotas that reset awkwardly, cold resumes measured in minutes. None of it is hidden — all of it is forgettable.
So now there's a runbook. It's short, almost embarrassing: graph-related ENOTFOUND across services → check instance state in the console → resume if paused → then check the retry backlog and dead-letter queues. Writing it felt like documenting how to turn a doorknob. But runbooks aren't for the day you learned the lesson; they're for the version of you eight months later, mid-incident, who once again knows it must be the connection pool.
Lesson three: outages are rehearsals
The genuinely valuable part of this incident is that it was a free rehearsal of a real failure mode: what happens to the system when one datastore disappears while the others keep running? I got to watch the blast radius with no users harmed. Some of it was reassuring — Pub/Sub held undeliverable work and retried, nothing was silently dropped. Some of it produced follow-up work, mostly around how quickly a human (me) finds out that a backlog is forming, rather than discovering it by tripping over the symptoms.
Staging outages are the cheapest chaos engineering a solo founder will ever get. The only way to waste one is to fix it without writing anything down.
The takeaway
If a managed dependency starts failing everywhere at once, look at the service's console before your own code — wholesale failure is an infrastructure fingerprint, not a bug fingerprint. Know the lifecycle quirks of the tier you're paying for, because the cheap tiers earn their price in behaviours you'll forget by the time they matter. And when the incident is over, spend the extra ten minutes on the runbook. My graph database will sleep again someday. Next time, waking it up is a checklist item, not a morning.