Players Engine

Devlog · Entry 16 · Engineering Notes

Structured logging on a budget

· Nic Vannetti · 5 min read

Nobody budgets for logging. It feels free — it's just printing, after all — right up until the first month your observability line item rivals your compute. Cloud logging platforms charge for ingestion and retention, which means every stray debug line in a hot path is a tiny recurring subscription someone signed without reading. On a bootstrapped platform, logs are a product: they have users (usually me, at the worst hour), they have costs, and they deserve the same design attention as any feature.

Structured or it didn't happen

Every service logs JSON through pino — one line, one event, with fields instead of prose. The win isn't aesthetic. Structured fields are what turn a log platform from a text search into a database: filter by request ID, group by route, chart error rates by service version. The moment logs are sentences ("Something went wrong updating the thing"), every investigation starts with regex archaeology. The moment they're records, investigations start with a query. On a team of one, the minutes that difference saves during an incident are the whole ballgame.

Context rides along automatically

The framework hands every request handler a child logger pre-loaded with request context — request ID, route, and the relevant entity identifiers. Handler code just says log.info(...) and the context arrives without anyone remembering to attach it. This is the logging sibling of a rule that appears throughout the standards: correctness by construction beats correctness by vigilance. If attaching context is manual, the one log line that matters during an outage is the one where somebody forgot.

Levels are cost dials, not decoration

The levels have meanings with teeth. error means a human should eventually look; it's wired to alerting, so noise here trains alarm-deafness. warn means the system absorbed something abnormal — a retry that succeeded, a fallback taken. info marks business events at a rate proportional to traffic, never per-loop-iteration. debug exists for development and is compiled out of production by configuration, because at cloud prices, per-iteration logging in a hot path is the most expensive prose I can write. When a mystery genuinely requires verbose logs in production, the level flips for one service, temporarily — configuration, not code.

Redact by default, not by memory

The logger is configured with a redaction list — authorization headers, tokens, email addresses, anything credential-shaped — that scrubs values before they leave the process. This is the second line of defence; the first is not logging whole request objects in the first place. Logging a request "for debugging" is how credentials end up retained for thirty days in a system with different access controls than the database those credentials protect. The redaction config means the mistake has to be made twice, in two different files, to matter.

Design for the reader at 3 a.m.

The test of a logging setup is the incident. What actually gets queried here, in practice: a request ID pulled from an error report, expanded into its full request trace across services; an event identity followed end-to-end through the async pipeline to answer "did the side-effects run?"; instance age filters to separate cold-start weirdness from steady-state bugs; and error groupings by route and version to answer "did the last deploy do this?". Every one of those queries works because the fields it needs were attached automatically, by construction, back when nobody was under pressure.

Logs feed alerts; alerts stay scarce

The error level is wired to alerting, which makes noise discipline a paging policy, not a style preference. The rule: alert on symptoms users can feel — elevated error rates on a route, an async pipeline processing nothing during active hours — and never on individual log lines that a retry will absorb thirty seconds later. Every alert that fires and requires no action gets one of two treatments: the threshold moves, or the log level drops. A pager that's wrong twice a week trains you to ignore it the one night it's right, and on a team of one there is no second reader to catch what alarm fatigue missed.

The ban

console.log is banned in production paths — not as style pedantry, but because it's the unstructured, unleveled, contextless version of everything above. It can't be filtered, can't be priced, can't be redacted, and can't be found when it matters. The one place raw console output survives is fatal startup errors, where the logger itself may not exist yet and the only job left is to die legibly. Everywhere else: if it's worth printing, it's worth printing as data.

Want in on a future wave?

Drop your email and we'll let you know when Players Engine opens up. Early supporters get first access.

← All devlog entries