The clean gap was one topic

I built a small thing this week that I have wanted for a long time. When I ask a question, before the model sees it, my own notes are searched and the matching one is placed into the conversation. Not a suggestion to go look. The note itself, already there.

The difference matters more than it sounds. Every version of this I had built before handed the model a pointer: you have notes on this, go fetch them. That works on a model that follows instructions and fails on one that does not, and it fails silently, because a model that skips the fetch answers from its own guess and sounds exactly as confident. Putting the text in the conversation removes the obedience step. There is nothing left to skip.

It needed a threshold. The search returns a similarity score, and somewhere there is a line: above it, the note goes in; below it, silence. Too low and every idle remark drags in a note that has nothing to do with anything. Too high and the feature does nothing.

So I did what I would tell anyone to do. I let it run in observe mode, collected ten real cases, and looked at the numbers instead of guessing.

They separated beautifully. Genuine matches at 0.70 and 0.79. Everything else in a band from 0.50 to 0.59, and that band was made of the small change of conversation: “yes but I did not understand why you did not check first”. A gap you could park a car in. I put the line at 0.65 and wrote a paragraph in the code explaining, with the numbers, why 0.65.

The next morning I tested it with a question about a decision I had recorded months ago, the rule for which model gets to see private material and which does not. It is written down. It is indexed. Three separate notes on the subject came back, all correct.

It scored 0.5974 and the system said nothing.

The clean gap was an artifact of my sample. All ten of those cases came from one evening’s work, and on the correct side of the split there was exactly one subject: where my backups are kept. That subject happens to score high. Every other genuine question I could have asked was absent, not because it would have failed, but because I had not asked it.

I widened the sample to twenty, this time deliberately varying what the true cases were about. The gap closed. Here is the middle of the ordering, and it is the whole finding:

A question about a decision on infrastructure, 0.6107, genuine. The privacy rule, 0.5974, genuine. “Yes but I did not understand why you did not check first”, 0.5814, noise. “Refactor this method so it is smaller”, 0.5753, noise. “Where do the repositories that run on a schedule live”, 0.5743, genuine and recorded and sitting below both of those.

They interleave. There is no line. At 0.65 the feature was admitting four of eight real questions, which is to say it was silent on exactly the thing it was built for. The best a single number can do on that sample is 0.59, which takes seven of eight and still admits none of the noise, and I moved it there while writing down that the eighth is a miss I am accepting rather than a problem I solved.

The reason I can accept it is that a miss and a false positive cost different things. When the system stays silent the model still has a standing instruction to go and look, so a miss costs one extra step. A false positive costs a wrong answer wearing my own handwriting, because the note arrives labelled as my record, and the model treats it accordingly. I saw that too: asked to explain a numerical algorithm, with a marginal note admitted, the reply ended with a paragraph connecting the algorithm to an unrelated note of mine about scoring. Fluent, plausible, and about nothing.

Two things I keep.

Before believing a separation, look at the variety of the positive side, not its size. Ten samples felt like enough because I was counting samples. One subject is one subject whether it appears once or forty times, and a threshold cut against it is fitted to a topic rather than to a boundary. The negative side had plenty of variety, which is exactly why the picture looked so convincing.

The second is that the number was never the real answer. Two genuine questions and two throwaway remarks are interleaved by similarity score, which means similarity is measuring something other than what I need. What separates them is structural. One is a question about my own recorded world; the other is conversational continuation, or an instruction to write code. I have a scorer built for that shape already, sitting in shadow mode elsewhere in the system, and it is the real fix. The 0.59 is a placeholder wearing a decimal point, and I have written that into the code so that the next person to read it, who will be me, does not mistake it for calibration.