Quarantine first: how a new test earns its place in the gate

Every new test is a promise: if this goes red, something is wrong with the product. A test that breaks that promise even occasionally does real damage. It blocks a deploy for no reason, someone spends an hour finding out it was nothing, and the next time it goes red, people are a little slower to believe it.

So in our design, no test goes straight into the blocking gate. Not the ones we write by hand, and especially not the ones generated from a confirmed finding. Every new test starts in quarantine. It runs on every deploy, its results are recorded, but it can't stop a release. It gets promoted only after it has earned it.

The interesting question is what "earned it" means.

Why ten clean runs is too weak

The usual rule of thumb is something like "if it passes ten times in a row, it's fine." It feels like a lot. It isn't.

Imagine a test that flakes 10% of the time. That's a bad test. In a gate of 150 tests, a few of those would block deploys constantly. What's the chance it passes ten runs in a row anyway?

0.9 ^ 10 ≈ 0.35

About one in three. A rule that lets a third of your worst tests through isn't much of a filter.

It gets worse in practice, because those ten runs are rarely independent. They're often on the same commit, on the same afternoon, against the same staging data, maybe even back to back in a retry loop. A test that only flakes when the nightly data import is running, or when a cache is cold after a deploy, can pass ten times in an hour and fail the next morning. Ten runs in a row on one commit mostly tells you the test is stable on that commit, at that time.

A sequential test instead

What we actually want to know is: is this test's flake rate low enough to trust, or high enough to worry about? That's a question statistics has a good answer for.

Abraham Wald's sequential probability ratio test (SPRT) was built for exactly this kind of decision. Instead of fixing the number of observations up front, you look at each new result as it arrives and ask whether the evidence is strong enough yet to decide. If it is, you stop. If it isn't, you keep watching.

We set it up with two hypotheses:

• the test is good: it flakes at most about 1% of the time
• the test is bad: it flakes about 10% of the time or more

and we accept a 5% chance of being wrong in either direction. Each clean run nudges the evidence toward "good" by a small, fixed amount. Each failure pushes it hard toward "bad", because a failure is far more likely from a flaky test than from a good one. With those settings, a test with no failures at all needs about 31 clean observations before the evidence is strong enough to promote it.

Compare that with the rule of thumb. The 10%-flaky test from earlier has roughly a 4% chance of getting 31 clean runs in a row, instead of 35% for ten. And because the test is sequential, a single failure along the way doesn't just reset a counter. It shifts the evidence so far that the test needs a long run of clean results to recover.

What counts as an observation

The math only works if the observations mean something, so we're strict about what counts.

The 31 clean observations have to be spread across several commits and several days. Many runs on one commit in one afternoon count for much less than the number suggests, for the reasons above. We want the test to have seen different code, different data, a cold cache and a warm one, a quiet morning and a busy evening.

Runs that failed because of infrastructure don't count either way. If the staging site was down or the browser couldn't start, that says nothing about the test. The system separates those failures from test failures before they reach the statistics.

One flip on the same commit

There's one more rule, and it applies to tests already in the gate, not just new ones.

If a gate test passes and then fails on the same commit, with no code change in between, it's marked suspect. The product didn't change. The test did something different anyway. That's the clearest flakiness signal there is, and we don't want to wait for the statistics to agree.

A suspect test still runs and still gets recorded, but it stops being able to block on its own until someone looks at it. Usually the fix is small: a missing wait, a hard-coded date, a test that assumed it would get the first product in a list that has no fixed order.

This is also why gate retries matter. With 150 gate tests and a 1% flake chance each, no retries means about 78% of deploys get falsely blocked by something. One retry brings that to about 1.5%. But a retry that turns red into green is exactly the "flip on the same commit" signal, so a retry never hides the problem. It just keeps the deploy moving while the test goes back under watch.

We've written before about how much flaky tests cost a team in Blue Monday, red builds. Quarantine is our answer to keeping them out of the gate in the first place. It's slower than "ten green runs and ship it." We think that's the right kind of slow.