Session-based test management, but for agents

The first time we pointed a model at a little demo shop we'd thrown together on a laptop, with the instruction "explore this site and find bugs", it did what an unsupervised intern on their first day might do. It clicked the logo. It opened the cookie banner. It read the privacy policy with great attention. Then it found the newsletter signup and tried eleven slightly different email addresses.

Nothing it did was wrong, exactly. It just had no idea what it was there for.

Human testers had this problem long before models did, and someone already solved it.

What the Bachs figured out

In 2000, Jonathan and James Bach published session-based test management. The problem they were facing was that exploratory testing worked, but managers couldn't see it. A tester would spend a day exploring and come back with "I tested the reports module." Tested it how? For what? How much of the day was actually testing, and how much was setting up data or chasing a broken environment?

Their answer was a small amount of structure wrapped around a lot of freedom:

• A charter: a short mission for the session, like "explore the export feature with large data sets, looking for truncation and encoding problems."
• A time box: a session lasts a fixed, uninterrupted stretch, often around 90 minutes.
• Session notes: what was done, what was found, what got in the way.
• A debrief: a short conversation afterward where a lead reviews the notes and decides what's next.

The tester still decides what to click. The charter just tells them why they're clicking. Elisabeth Hendrickson's "Explore It!" (2013) builds a lot on the same idea, and her charter template ("Explore target with resources to discover information") is pinned on more than one of our screens.

Mapping it onto agents

Almost every piece of this carries over, with a few honest adjustments.

The charter comes from the planner. Before any explorer starts, a planner reads the diff and the tickets for this deploy. If the change touched the coupon logic, a charter might read: "Explore checkout with stacked coupons and currency changes, looking for totals that don't add up." The explorer gets that mission, a persona and a starting page. Not "explore the site."

The time box is a step and cost budget. A model doesn't get tired, but it does get expensive, and it does wander. So a session has a hard limit on browser steps and model spend. When it runs out, the session ends, whatever it was in the middle of. That forces the planner to write charters that are small enough to finish.

Session notes are structured. Every step is recorded: the action, the URL, a screenshot, the network calls. If the explorer thinks it saw something, it doesn't write "the total looked wrong." It files a runnable condition with the steps to get there. We've written about why in Why the LLM should suggest, and code should decide.

The debrief is split in two. This is the biggest change. In the original method, the debrief was where a human looked at the findings and judged them. For agents, the judging part goes to a referee that replays each finding in a clean browser with no model involved. The planning part (what should the next session look at?) goes back to the planner. The explorer itself never decides whether its own finding is real.

That last rule matters more than it sounds. An explorer that also judges its own findings has every incentive, in whatever sense a model has incentives, to believe them.

What we're still unsure about

A few things don't map neatly.

Human sessions produce something hard to measure: the tester learns. The next session is better because the person remembers what felt fragile. An agent session starts fresh unless we hand it that memory. We're experimenting with giving each charter a short summary of past sessions on the same area, but we don't yet know how much context helps before it starts to bias the explorer toward old paths.

The Bachs also tracked how a session's time was split between testing, bug investigation and setup. We'd like an equivalent for agents, something like "steps spent exploring vs. steps spent recovering from a login timeout." It would tell us when the environment, not the explorer, is the problem.

And charters are only as good as the planner writing them. A vague charter produces the same aimless newsletter-signup tour as no charter at all.

If you've run session-based testing with a human team, we'd love to know what made your charters good. We suspect the answer applies to agents too. For more on where our thinking here comes from, see The testers who taught us how to explore.