A confidence score you can calculate

Ask a language model how confident it is that something is a bug and it'll happily tell you. "92% confident." "High confidence." It sounds precise. It isn't anchored to anything.

A model's self-reported confidence is just more generated text. It isn't calibrated against how often that model is actually right about bugs, and in published agentic testing benchmarks, free-text bug verdicts are right roughly one time in four. A "92%" stapled onto a one-in-four guess doesn't make the guess better. It makes it more persuasive, which is worse.

So when we designed the confidence score that decides where a finding goes, we threw the model's opinion out completely. The score is computed from things we can count.

Three numbers, multiplied

confidence = oracle weight × reproduction score × rule precision

Each part answers a different question.

Oracle weight: how trustworthy is the thing that said "wrong"? We rank oracles from strongest to weakest. A specified expectation ("this page must show the VAT line") or a universal invariant (a 5xx, a JavaScript error, a broken image) is about as solid as it gets. A business invariant like "in-stock products have a price above zero" is close behind. Consistency checks, metamorphic relations and statistical oracles get progressively lower weights. Agent judgment on its own sits at the bottom, low enough that it can never climb out of "suspect" by itself.

Reproduction score: how often did it happen again? The referee replays the finding in a clean browser session with no model involved. If it reproduced k times out of n attempts, the score is:

reproduction = (k + 1) / (n + 2)

This is Laplace's rule of succession, and we picked it because it behaves sensibly with small numbers. A raw k/n would say that 1 out of 1 is 100% certain, which is silly. With the +1 and +2, one success out of one attempt gives 2/3. Three out of three gives 4/5, or 0.80. Five out of five gives 6/7, about 0.86. It rewards repetition, but it never pretends a handful of replays is proof.

Rule precision: how often has this rule been right before? Every rule keeps a history. When one of its findings is confirmed as a real bug, that counts in its favor. When a human marks it as a false positive, or as technically true but not worth fixing (the "effective false positive" idea from Sadowski and colleagues' work on static analysis at Google), that counts against it. A new rule starts with a modest prior and earns its way up.

Where a finding goes

Once we have the number, routing is mechanical:

• 0.85 or above goes to Jira automatically, with repro steps, trace and screenshots attached.
• 0.50 to 0.85 waits in an approval queue for a person to say yes or no.
• Below 0.50 stays a "suspect." It's kept, it's visible, but nobody gets paged over it.

Notice what the multiplication does. A finding needs to be strong on all three axes to clear 0.85. A business-invariant finding with a high oracle weight, a rule with a long clean history, and a reproduction score from several clean replays can get there. The same finding that only reproduced 2 out of 3 times lands at 3/5 on reproduction, and even with perfect marks elsewhere it goes to a human first. An agent-only observation with no backing rule never gets near the top, no matter how many times it reproduces, because its oracle weight caps it.

That's on purpose. Automatic filing is a privilege, and it should be hard to earn.

Critical things still get a human

There's one exception to "let the number decide." If a finding is critical, say checkout returns a 500 or prices show as zero across a whole category, it goes where the score sends it and a human gets notified right away. A critical finding at 0.72 shouldn't sit quietly in a queue until someone happens to open the dashboard on Monday.

The score decides routing. It doesn't decide whether someone should know.

Why bother with all this

Because every number in the formula can be explained. When a developer asks "why did this land in my Jira at 3 a.m.?", the answer fits in a sentence. Something like: the rule is "no duplicate products in a listing," its history says it's almost always right, and the referee reproduced it every time it tried. Nobody has to trust a vibe.

We're getting ready to point all of this at a real staging site soon, and that's when the rule precision numbers will start meaning something. Until then they're priors. We're curious which of them will survive contact with real data.