The outage nothing inside could see

I have a set of background jobs that keep my notes in order overnight. They link related ideas, judge which surfaced notes were useful, check whether old claims still hold, take backups.

While I was travelling, the power went out at home. The machine came back up when it returned, the daemon restarted, the jobs resumed on schedule. Nothing broke. Every health check I have was green, and every one of them was telling the truth.

Three nights of work simply had not happened.

This is the part that took me a while to accept as a design problem rather than bad luck. All my monitoring asks the same shape of question: did this thing fail? A backup that errors, a job that exits non-zero, a model call that times out. Every one of those checks is looking for an event. The absence of events produces nothing to look at.

And the absence is invisible from the inside, specifically. The process that would notice was not running. When it started again it had no memory of the gap, because its notion of time is the last run it recorded, and the last run it recorded was three days ago and completed successfully. From inside the system, nothing distinguishes “I ran last night” from “I ran seventy hours ago and the clock moved”.

The fix is dull and I like it for that. The morning briefing now carries a line saying how long it has been since the automation last did anything, whenever that gap is longer than expected. Not an alert. Not a failure. A sentence in the thing I already read, saying the last time this system did any work for you was Sunday.

There was a second half I got wrong at first. When the machine woke, all the missed jobs fired at once, and they compete for the same hardware. The catch-up itself became a small stampede, which is its own way of turning an outage into a second incident. So the scheduler now runs at most a few of them at a time.

What I keep from it: I had built a system that could tell me when it was wrong and not when it was absent, and those are different questions. Failure produces a record. Silence produces nothing, which means the only way to see it is to measure the shape of the nothing, deliberately, from somewhere that kept running.

The nearest human version is a smoke alarm with a flat battery. It is not lying to you. It is not telling you anything, and the two feel identical from the sofa.