Configuration bugs are the dumbest class of outage. Nothing is architecturally wrong, no algorithm failed — a string was missing, or present but pointing at the wrong universe. Across a fleet of small services, the surface area for that dumbness multiplies, so configuration is one of the few places where this project is genuinely dogmatic. The rules are simple enough to fit on an index card, and each one was purchased with a bad afternoon.
Rule one: the service refuses to start half-configured
Every service validates its entire environment at startup, before binding a port. Required variables present? Formats plausible — URLs parse, numbers are numbers, enums are members? If not, the process exits with a message naming exactly what's wrong. It does not start "mostly working" and then explode at the first request that touches the missing value — which, by Murphy's law, is a real user's request at a time you're asleep. A service that can't be configured correctly should be a service that visibly isn't running. Deploy pipelines catch that; 3 a.m. doesn't have to.
Rule two: one config module, or chaos
Reading the raw environment is allowed in exactly one file per
service — the config module that validates everything and
exports a typed object. Nowhere else does
process.env appear. This sounds like bureaucratic
fussiness until you've grepped a codebase where any file might
quietly consult an environment variable: you can never again
answer "what does this service need to run?" with confidence.
With a single choke point, the config module is the
documentation — complete, typed and incapable of drifting from
reality, in the spirit of the
standards page.
Rule three: secrets never have defaults
The most dangerous line of code in this category looks helpful:
a signing key that falls back to "dev-secret" when
the real one isn't set. Now the failure mode isn't a crash — it's
a service that runs happily while accepting tokens anyone on
GitHub can forge. A missing secret must be a fatal error, never a
silent downgrade. Corollary: secrets live in a managed secret
store in production, injected at deploy time — never in the
image, never in the repo, and never echoed back by any logging
or diagnostics path. The
logging setup
redacts the usual suspects by name as a second line of defence.
Local and staging: same shape, different values
Local development uses env files that are git-ignored, documented by a committed example file listing every variable with placeholder values — so a fresh clone knows exactly what to provide without being handed anything sensitive. Staging gets real values through the same deployment mechanism as production, because configuration paths you don't rehearse are configuration paths that fail when it matters. The invariant across all three environments: identical variable names, identical validation, different values. The moment "works locally" and "works deployed" diverge, the difference should only ever be a value — never a mechanism.
Build-time values are a different species
One distinction that saves real confusion: some "environment variables" aren't runtime configuration at all — frontend frameworks inline certain values into the compiled bundle at build time. Those obey none of the rules above. Changing them means rebuilding, not redeploying; validating them at server startup validates nothing; and they are public by definition the moment they ship to a browser, so nothing secret may ever travel that path. Our conventions mark them explicitly, because a value that looks like config but behaves like code is exactly the kind of ambiguity that ends in a leaked key or a "why didn't my change apply" evening.
The unglamorous part: rotation
Everyone's system handles rotation beautifully in the design doc. In practice, rotating a credential means knowing every consumer of it — which the one-config-module rule finally makes knowable — and having the patience to stage the swap: add the new value, deploy, confirm, retire the old one. The honest admission is that rotation here is still a manual checklist, not an automated dance. What made it tractable was reducing the number of long-lived secrets worth rotating at all: deployment credentials are already short-lived and keyless, as covered in the Workload Identity entry, and each retired static key is one less 3 a.m. regret waiting to happen.
The card, in full
- Validate everything at startup; refuse to boot otherwise.
- One config module;
process.envnowhere else. - Secrets have no defaults, ever.
- Managed secret store in prod; ignored env files locally; committed examples with placeholders.
- Same names and validation in every environment.
- Prefer credentials that expire over credentials you must remember to rotate.
None of it is novel. All of it is the difference between configuration being boring — the correct amount of exciting for configuration to be — and configuration being the reason your weekend ended early.