Field notes: our first dry run against a live staging site

For nine months everything we built lived on our own machines and never touched a real site. Today it did. We ran it against the staging site of a multi-vendor B2B marketplace we partnered with.

We want to write this down while it's fresh, including the parts that didn't go well.

The setup, such as it was

None of the infrastructure we've been describing on this blog was in place. No dedicated server, no dashboard, no Jira integration. One of us drove both runs from a laptop over the partner's VPN.

That sounds like a disclaimer, and it is one. It also turned out to be useful. With nothing hiding the moving parts, every rough edge was right there on screen.

We did two dry runs. The first used one model. For the second we wanted Claude Opus 5.5, and for a while it looked like we couldn't have it: every call came back as if the model wasn't available. It took longer than we'd like to admit to see that we had loaded the wrong API key. Nothing was wrong with the model. We had the wrong credentials. We fixed the key and ran again.

The second exploratory session took 34 steps and cost about $2.60.

What the referee confirmed

Five rule findings came out of the runs. The referee replayed each one in a clean browser session with no AI involved, three times. All five reproduced 3 out of 3.

1. Infinite scroll repeated products. The category listing showed 147 product cards, but only 144 unique products.
2. Ten in-stock products at a price of 0.00. In stock, purchasable, and apparently free.
3. The "lowest price" sort ordered by the undiscounted price. A product on sale could show a lower price than the one listed above it.
4. Two broken images.
5. A JavaScript error on the homepage, caused by a banner being initialized twice.

None of these is dramatic on its own. Every one of them is the kind of thing a customer notices before a developer does, and none of them would fail a typical end-to-end test that clicks through checkout.

The 147 and the 144

The duplicate products are the finding we've talked about most today, because they show where the line between agent and referee sits.

The rule is simple: a listing shouldn't show the same product twice. The referee checked it with code. Count the cards, count the unique product IDs, compare. 147 against 144. Confirmed, three times out of three.

The agent went further and suggested why. The first render showed 27 cards. When you scrolled, the ajax call asked for page 2 with a page size of 24. Page 2 at size 24 starts at item 25, so items 25, 26 and 27 showed up a second time. Three overlapping items, which is exactly the gap between 147 and 144.

It's a good explanation. We think it's probably right. But the referee can't verify a theory about pagination by itself. It can verify that duplicates exist. So the explanation doesn't become part of the finding. It becomes a note on the ticket, clearly marked as the agent's suggestion, for a developer to confirm or throw away. That's the LLM suggests, code decides in practice, and we were glad to see the split hold up on a real site.

Replaying with no model at all

Later, we replayed the earlier findings again, this time with no model anywhere in the loop. Just the recorded steps and the rules.

5 out of 5 reproduced, at zero model cost.

This is the result we care about most from today. It means a finding, once proven, doesn't need the agent anymore. It can become a check that runs on every deploy for free.

What we got wrong

Of the 34 steps in the second session, 3 were wasted on tool errors rather than testing:

• The agent tried to act while the page was still navigating, and the action hit a page that was halfway gone.
• It looked for the sort dropdown by its label instead of its role, didn't find it the way it expected, and had to try again.
• A click timed out.

Three steps out of 34 isn't a disaster, but it's almost a tenth of the session spent fighting the browser. All three are fixable in our tool layer: wait for navigation to settle before handing control back, prefer roles when locating controls, and handle slow clicks more patiently. That's this week's work.

The second lesson is about reporting. The agent independently reported some of the same problems the rules had already caught. Filing those separately would mean two tickets for one bug, and someone on the partner's team closing one as a duplicate. From now on, when an agent finding overlaps a rule finding, it gets attached as a note on the rule finding instead of standing on its own.

What's next

A real server, so this doesn't depend on one laptop and one VPN session. A dashboard, so people other than us can see the evidence. Jira, so findings land where the partner's team already works. And the three tool fixes above, before the next run.

It was a small run. It was also the first time the whole loop touched something real, and it mostly held.