Why the LLM should suggest, and code should decide

Ask a language model "is this a bug?" and you'll get a confident answer. The trouble is that the confident answer is wrong more often than it's right. Across published benchmarks for agentic UI testing, a free-text "this is a defect" verdict lands somewhere around one in four. Build a product on top of that number and you don't get a tester. You get a very eloquent source of noise.

So we're building Testomatica on a split that sounds almost boring until you see what it prevents: the LLM suggests, and code decides.

Two jobs that should never be the same job

There are two very different questions inside "testing":

• What should we look at? This is open-ended, creative, and context-heavy. It benefits enormously from a model that can read a page, imagine a user, and wander somewhere a script never would.
• Is what we saw actually wrong? This needs to be exact, repeatable, and defensible. The same input must always produce the same verdict, or the result is worthless as a signal.

The first is a great fit for an agent. The second is a terrible fit for one. Most AI testing tools collapse the two and let the model do both, which is exactly where the false alarms come from.

Flow in code, judgment in agents. Orchestration, budgets and environment safety are deterministic. The agent only decides what to explore, never what counts as broken.

Findings arrive as conditions, not opinions

When an agent spots something suspicious, it doesn't file "the price looks wrong." It files a runnable condition, something like card_price == product_page_min_price, along with the steps to reach it. That condition is then evaluated by code, not by another prompt.

This one rule changes the economics of the whole system. A verdict that a human can read, a machine can re-run, and a test can inherit is worth a hundred paragraphs of model prose.

Then it has to reproduce

Even a condition can fire on a fluke. So before anything reaches your issue tracker, a separate referee replays the exact steps in a clean browser session with no model in the loop. The rule of thumb: reproduce it at least twice out of three before calling it real. Intermittent noise mostly washes out; genuine bugs mostly survive.

Why this matters for teams shipping fast

If you're merging AI-written code several times a day, the last thing you need is a tool that cries wolf. A false-alarm rate above twenty percent trains everyone to ignore the tool, and an ignored tester is worse than no tester because it gives you a false sense of safety.

Keeping the model on the suggesting side of the line, and code on the deciding side, is how you get the creativity of an agent without the chaos. Next we have to prove it works on a real site. That part is still ahead of us.