An infrastructure failure is not a test failure

It's 2 a.m. and a test run goes red. Forty tests failed at once. Login, search, cart, checkout, all of them, within the same second.

Nobody broke forty features at once. The VPN dropped.

Anyone who has run a test suite for long has seen this. And the dangerous part isn't the failed run. It's what happens if that run gets treated like a normal failure: forty tickets in the tracker, a blocked deploy, and a team that learns, one more time, that red doesn't really mean red.

Two kinds of red

We're designing Testomatica around a hard split between two outcomes:

• Test failure: the product did something wrong. A price is 0.00, an image is broken, the sort order is off.
• Infrastructure failure: we couldn't find out. Something between us and the product, or inside our own setup, stopped working.

The second kind isn't a softer version of the first. It's a different thing entirely, and it gets handled by different code.

Here's what counts as infrastructure in our design:

• VPN down. Staging often lives behind a VPN. If the tunnel is gone, every request fails, and that says nothing about the site.
• Model provider outage. Explorer agents call a model. If the provider is returning errors or timing out, exploration can't happen. The product is fine; our dependency isn't.
• Site unreachable. DNS fails, the staging host is being redeployed, a load balancer is restarting. We can't test what we can't reach.
• Budget exhausted. The AI budget for the run or the day is spent. Agents stop. That's a decision we made, not a bug someone shipped.

How we tell them apart

The first defense is preflight. Before any test runs, we check that the site responds, that access works, and that there's budget left. If preflight fails, the run doesn't start, and it's recorded as an infrastructure problem.

But things also break mid-run. So every failure gets a quick look before it's classified. If one test fails on a timeout, that's a candidate test failure. If every test fails within the same few seconds with connection errors, that pattern is infrastructure. If the model API returns a 503, that's not something the site did.

The key rule: infrastructure failures are never filed as bugs. Not in Jira, not in the "suspect" queue, nowhere a developer would be asked to fix the product. They're logged, shown on the dashboard in their own color, and the run is retried when things recover.

Where it matters most is the release gate. If we can't test, we don't say "go." We also don't say "stop, bugs found." We say we couldn't decide, and the call goes to a human.

When it doesn't recover

Short blips are easy. The VPN reconnects after a minute, the run retries, everyone sleeps.

Long outages are different. If staging has been unreachable for an hour, someone needs to know, but it's not the developer who last merged a pull request. So a long infrastructure outage opens an infra ticket: one ticket, clearly labeled, saying what failed, since when, and what we tried. It goes to whoever owns the environment, not into the product backlog.

One ticket, not forty. And never one that says "checkout is broken" when checkout is fine.

Why this is worth the effort

It sounds like plumbing, and it is. But the value of a test system is entirely in whether people believe it. Every time an outage gets reported as a product bug, belief goes down a bit. Do that a few times and the real bug, the one that shows up the next week, gets dismissed as "probably the VPN again."

We'd rather spend the effort up front to make red mean exactly one thing.