It judged nothing and reported success

Sapix runs a nightly pass that decides which of my notes deserve to be linked. The judging is done by a small local model, because the candidates come from my personal graph and that material does not leave the machine. One morning I checked the night and found a perfect crime scene: the pass had walked all 2,500 candidate pairs, judged exactly zero of them, exited with code 0, and stamped its own checkpoint as a successful full scan. The summary line said errors=0, and technically it was telling the truth.

The model server was up. It answered the port, it answered health checks. It just had no model loaded, because a hardware experiment days earlier had swapped the builds. And the pass treats an unavailable judge as “skip this pair”, by design: when the judge cannot rule, guessing would poison the graph, so skipping is correct. Every skip was correct. The sum of 2,500 correct skips was a night of nothing wearing the costume of a night of work.

There was a second trap underneath. This local server routes a request for a stale model name to whatever model happens to be loaded, and only errors when nothing is loaded at all. So a wrong id in the config does not fail; it silently runs on whatever is resident. A misconfiguration that cannot fail loudly is a misconfiguration you will find weeks later, by accident.

The repairs were boring on purpose: load the model, reset the burned checkpoint, fix the ids everywhere they lived. The interesting part is what gatekeeps the repair. The replacement build was not the one I had validated, so before letting it near the graph I re-ran the fixed golden set, twelve cases, three of them traps, at temperature zero. Twelve of twelve, zero trap hits. Only then did the night get its judge back. Without a golden set, “quality is inviolable” is a slogan; with one, it is a procedure.

What changed permanently: summary lines now carry the denominator. A line that says judged 0 of 2,500 screams; a line that says errors=0 purrs. Exit codes measure whether the process survived, not whether the work happened. And silence is only a signal when something is guaranteed to speak.