Every microservice in PlayersEngine has unit tests. They run fast, they're deterministic, and for a stretch of this year they were lying to me. Not individually — each service really did do what its tests claimed. The lies lived in the seams: the places where one service's assumptions about another service stopped being true. The only thing that reliably catches those is running the real request against the real stack, so that's what I do now, before every merge that touches a user-facing flow.
Where the bugs actually were
Three examples from recent memory, all found by end-to-end sweeps and none findable by unit tests as written.
A malformed graph query that only the real database rejects. One service builds Cypher queries against Neo4j. The unit tests mocked the driver — reasonable, standard practice — and asserted the query text looked right. The real database parsed the query and threw a syntax error the mock never would have, because the mock doesn't implement Cypher. Every request down that path was a 500 in staging while the test suite glowed green.
A PATCH that 500s only after a prior write. An update endpoint worked perfectly against a fresh fixture. But if you called it after an earlier write had already shaped the document — the normal sequence for a real user — a field collision made it blow up. Unit tests construct their own preconditions, and I had constructed the wrong one. The E2E sweep, which performs the same sequence a person would, hit the broken ordering immediately.
A frontend fixture that didn't match reality. The webapp's tests fed a component data shaped the way the frontend believed the API responded. The API had evolved. Both test suites passed, because each side was testing against its own copy of the contract. The seam — the actual JSON crossing the wire — belonged to neither suite.
The pattern is consistent: unit tests verify each service's model of the world, and integration bugs are precisely the places where two models disagree. More unit tests don't help; they just make each side more confident in its own wrong model.
What my E2E setup actually is
Nothing fancy, deliberately. A set of scripts that exercise real user flows against staging: mint a real auth token for a test account, call the public API gateway the way the browser would, assert on responses and on the resulting state. The real gateway, the real Firestore, the real Neo4j, the real Pub/Sub pipeline with its worker consuming events.
Two properties matter more than tooling choices. First, go through the front door. Testing services directly bypasses the gateway config — auth enforcement, header forwarding, route definitions — which is itself a rich source of production-shaped bugs. A route that works when called directly but has a missing security stanza at the gateway is broken in every way users care about, and only front-door tests see that. Second, assert on side-effects, not just responses. In an event-driven system, a 200 means the request was accepted, not that the work happened. Where a flow enqueues an event, the test polls for the downstream state change. That's how you catch the queue with no subscribers — an entire class of silent failure where everything returns success and nothing happens.
The case for shared, mutable, imperfect staging
The purist answer is hermetic environments: spin up a fresh, isolated stack per test run, seed it, tear it down. I understand the appeal and I don't do it. A hermetic copy of this platform means provisioning a graph database, a document database, a pub/sub emulator, a dozen services, and a gateway per run — real money and real maintenance for a solo founder, and the emulators reintroduce exactly the fidelity gap I'm trying to close. The Cypher bug above doesn't reproduce on a mock, and some of it wouldn't reproduce on an emulator either.
So my E2E tests run against the one shared staging environment, and I accept the classic costs: tests must tolerate pre-existing data, test accounts accumulate history, and occasionally a sweep fails because staging itself is mid-deploy. In exchange, the environment under test is the same one that mirrors production config — same auth path, same event topology, same database engines with the same error surfaces. When staging drifts from production intent, that drift is a bug I want tests to trip over. A hermetic suite would have happily passed while staging sat broken.
The discipline that makes shared staging workable: tests create their own entities where possible, use dedicated test accounts where not, and every assertion is written against state the test itself established. Flakiness gets investigated, not retried into silence — a flaky E2E test is usually a real race in the product.
Keeping it cheap enough to actually run
A safety net you skip is decoration. The sweeps stay under a few minutes by testing flows, not permutations: one pass through each critical journey, not a matrix of edge cases — edge cases stay in unit tests where they're cheap. Scripts are plain and dependency-light, so there's no test-framework upgrade treadmill. And because staging scales to zero, the marginal infrastructure cost of a sweep is close to nothing; the first request eats a cold start and I simply don't care in a test context.
The payoff compounds in a specific way: every latent 500 the sweep flushes out is a bug a beta tester didn't hit. With a small tester pool, every individual's first impression is expensive — a tester who hits three broken flows in a night doesn't come back. The E2E sweep is cheap insurance priced against exactly that.
The takeaway
Unit tests verify components; your users experience seams. If your architecture is services talking through a gateway and a message bus, some of your bugs structurally cannot exist inside any one service — so some of your tests structurally cannot be unit tests. Run the real flows against the real stack, through the front door, asserting on side-effects. It doesn't need to be elegant. Mine is a pile of scripts, and it has caught more user-facing breakage this year than every mock in the monorepo combined.