A gaming profile is only as interesting as its numbers are fresh. If someone links their game accounts and their rank still shows last month's tier, the profile is lying, and players notice immediately. So PlayersEngine has to keep stats from a growing list of third-party game platforms reasonably current. The obvious way to do that is a scheduled job. I don't use one, and the reasoning behind that choice is one of the more useful cost lessons of this build.
The cron job that scales with the wrong number
The scheduled-sync design writes itself: every night, walk every linked account, call the upstream API, store the result. It's easy to reason about and easy to monitor. It also has a cost curve tied to the wrong variable. Its bill grows with linked accounts, while the value it produces grows with profiles people actually look at.
Those two numbers diverge fast. On any social platform, attention follows a steep power law: a small slice of profiles gets most of the views, and a long tail gets almost none. Someone who linked an account once and hasn't logged in since still costs a nightly upstream call, a database write, and a slot in your rate-limit budget. At a million users with a few linked accounts each, that's millions of calls a night, most of which refresh numbers nobody will ever read. Some of those upstream APIs charge per call; all of them rate-limit. A nightly sweep also creates a daily traffic spike that you have to provision for, which works against a scale-to-zero budget that assumes load follows real users.
Refresh when someone looks
The alternative flips the trigger. Stored stats carry a timestamp. When a profile is viewed, the read path compares that timestamp to a staleness threshold. If the data is fresh enough, it's served as-is and nothing else happens. If it's stale, it's still served as-is, and a refresh gets queued in the background. The next view sees the new numbers.
This is stale-while-revalidate, the same pattern HTTP caches use, applied to a database of third-party data. Its key property is that upstream traffic now tracks attention instead of inventory. A dormant account costs nothing. A popular profile refreshes about as often as its threshold allows, no matter how many people view it, because every view after the first finds the data fresh again. Cost becomes a function of distinct stale profiles viewed, which is bounded by real engagement.
The price you pay is one slightly stale view: the first visitor after the threshold passes sees the old numbers. For stats that change over hours or days, that's an easy trade. I'd rather show yesterday's rank quickly than make the page wait on a slow third-party API, and the page never blocks on an upstream outage either.
Not every viewer needs the same freshness
Once refresh is triggered by a view, it's natural to ask whose view it is. The account owner looking at their own profile is the most freshness-sensitive audience there is: they just played, and they want to see the result. A stranger browsing is far more tolerant, because they have no idea what the number was an hour ago. So the threshold depends on the viewer: tighter for the owner, looser for visitors. A manual "refresh now" button gets its own cooldown so it can't be spammed against a paid API.
This single decision cuts upstream volume noticeably, because visitor views vastly outnumber owner views. It also puts freshness where people actually perceive it, which is what users care about, rather than applying uniform freshness everywhere.
Making lazy refresh safe
Triggering work from the read path has a few failure modes, and each needs a specific guard:
- Stampedes. A popular stale profile gets fifty views in the same second. Without a guard, that's fifty upstream calls for the same data. The fix is a short-lived "refresh in flight" marker, written atomically, so only the first view enqueues work and the rest just read.
- Reads that write. The view path must stay a cheap read. It never calls the upstream API itself; it publishes a message and returns. The actual fetch runs in a background worker, where idempotent, retry-safe handlers absorb duplicates and transient failures without the user ever waiting.
- Upstream budgets. Background refreshes still go through the same per-provider rate limits and lockouts as everything else that touches a paid API. Lazy refresh reduces volume; it doesn't make the budget optional.
- Failure backoff. If an upstream call fails, the timestamp still moves forward on a shorter retry window. Otherwise a broken provider turns every view into a failed refresh attempt.
Where schedules still belong
This isn't an argument against cron jobs in general. Data that everyone reads, such as a shared catalog or a leaderboard, is a better fit for a scheduled build: you do the work once and every reader benefits, often from a static file on a CDN at close to zero marginal cost. The rule I use is about fan-in. When one computation serves many readers, schedule it. When each computation serves one subject's readers and most subjects have none, do it lazily.
The takeaway
Before you write a scheduled sync, check what its cost scales with. If it's the size of your table rather than the attention your product gets, you're paying to refresh data nobody reads. Serve what you have, refresh in the background when someone actually looks, set freshness per audience, and put guards on stampedes and upstream budgets. The data stays fresh where it matters, and the bill tracks your engagement instead of your total number of users.