Nobody Watches the Night Shift
Operations · July 10, 2026 · 8 min read
An interactive system has a built-in detector: users. When a page breaks, somebody notices within minutes and says so. The feedback loop is imperfect and noisy and it is still the fastest failure detection most organisations own.
Background work has no such loop. When a nightly synchronisation stops running, nothing turns red. There is no request to fail, no user staring at a spinner, no error rate to climb. The system produces exactly the output it produced when everything was fine: nothing, quietly. The failure is discovered weeks later, sideways, when somebody asks why a report looks wrong.
That asymmetry should shape how background work is instrumented, and usually it does not, because the same dashboards get copied over from the request-serving side.
The primary question is “when did this last succeed?”
Most background monitoring measures the wrong noun. It measures runs: how many started, how many errored, how long they took. Those are properties of executions that happened. The dangerous condition is an execution that did not happen, and it appears in none of them.
The measurement that catches it is staleness — the time since the last successful completion, per job, per data set, per tenant. It is one number, it is always available, and it goes wrong in exactly the way the system does. A job that stopped being scheduled, a consumer that lost its subscription, a queue that was silently emptied, a worker pool scaled to zero by a misconfiguration: all of them show up as a staleness value that keeps climbing while everything else on the dashboard stays serene.
The corresponding alert is an absence alert, and it needs to be defined per unit of work rather than globally. “Some jobs ran in the last hour” is satisfied by the healthy ones while the broken one is silent. The check has to enumerate what should have run and compare.
That enumeration is the hard part in real systems, because the list of expected work is often implicit — spread across schedule definitions, feature flags and tenant configuration. Making it explicit is worth the effort. A registry of expected work with its acceptable staleness is the artefact that turns background monitoring from anecdote into something checkable.
Lateness beats depth, again
For queue-driven work the same argument applies with a different metric. Queue depth is a snapshot of how much is waiting; it says nothing about whether anything is moving. A depth of a few hundred could be a healthy system with brief waits or a stalled one with none at all.
Age of the oldest unprocessed item resolves it in one number. If that age is rising steadily, consumption has stopped or fallen behind production, and it does not matter which — the user-visible consequence is identical.
The subtlety is per-partition measurement. In partitioned systems the aggregate can look fine while one partition is entirely stuck, because the others keep the average down. Whatever the unit of independent progress is — partition, shard, tenant, priority lane — that is the unit the age metric has to be reported at.
Errors that nothing raises
Background systems produce a category of failure that never becomes an exception. The job ran, processed zero records, and exited successfully, because the query that should have returned work returned nothing after a schema change. The message was consumed, deserialised into a structure with all fields empty, and written. The batch completed with every item skipped by a filter that is now too aggressive.
None of these throw. All of them are wrong.
The defence is asserting on outputs rather than only on exit status. Expected volume ranges, invariants that must hold after a run, reconciliation between what was consumed and what was produced. A job that normally processes a substantial number of items and today processed none has told you something important, and only an output assertion will hear it.
This is the background-work equivalent of a health check that tests a real dependency rather than returning “OK” from a handler. Both distinguish “the process is alive” from “the process is doing its job”.
Tracing across the boundary
Debugging a synchronous chain is mostly solved: the trace shows the path. Async work breaks the chain at every queue boundary, and the default is that the producer’s trace ends where the message is enqueued and a new, unrelated trace begins in the consumer.
Propagating trace context through the message envelope reconnects them, and it is a small amount of plumbing with an outsized payoff. When it is missing, the question “why did this customer’s order not produce a shipment” requires reading several services’ logs and correlating by identifier and time, which is an afternoon. When it is present, it is a query.
The related habit is logging the identity of the work item, not just the run. A log line saying a batch failed is far less useful than one naming which items were in it, because the first question after any batch failure is which work needs replaying.
Retries hide the trend
A system with retries reports success after transient failure, which is the point. It also means the failure disappears from the record. If the first attempt succeeds today and the third attempt succeeds next month, both are green, and the degradation between them is invisible.
Recording attempts separately from outcomes — how many items succeeded first time, versus after retry — surfaces the slope. A rising retry rate is an early signal of a dependency getting worse, and it usually arrives well before the retries stop being sufficient.
What a useful background dashboard contains
Not run counts and durations, or not only. Per unit of work: time since last success, age of oldest pending item, first-attempt success rate, and the size of the dead-letter store with the age of its oldest entry.
That last one deserves a note. Dead-letter queues are where undeliverable work goes to be examined later, and in most organisations “later” is never. An unattended dead-letter queue is a record of work that was accepted and then dropped, and its oldest entry is a good proxy for how long the organisation has been fine with that.