The feature that passed every test and was dead anyway

Sapix has a mechanism that injects your hard rules into every session automatically, so the assistant starts already knowing the things you have said must always hold. It shipped with its unit tests passing and a smoke test passing. It was completely inert.

For days, the rules it was meant to surface simply never appeared. The session started, the rules stayed buried. No error, no failure, nothing red. Just silence where there should have been content.

Here is what was actually wrong. A step in the refinement pipeline dropped the source tags when it built the internal record. The query that selected the hard-rule notes matched on those tags. With the tags gone, it matched nothing, every time.

The unit tests fed records that already had the tags attached, so the query found them and went green. The smoke test ran the path end to end but not against the real shape of the data, so it also passed. Neither could see the hole, because the hole only existed where real data met the real query, and nothing in the suite stood at that exact spot. It surfaced in the cold morning run, on the real vault, days after I shipped it. The fix was one line carrying the tags through, then a backfill of the notes that had already lost them.

The lesson is about layering. Three kinds of check catch three different classes of bug, and none of them stands in for the others. Unit tests prove the logic is right against mocks. Smoke tests prove the path runs end to end. Real use proves the integrated system does the right thing on real data. A green suite tells you the code does what the tests imagined, which is not the same as what the world hands it.

So I stopped calling a change done on a checkmark. I want to watch it do the right thing, once, on the real thing, before I believe it.