Blue Monday, red builds: the real cost of flaky tests

Today is Blue Monday, the third Monday of January, supposedly the gloomiest day of the year. The idea came out of a marketing campaign rather than any real science, so feel free to ignore it.

Engineers have their own version, though, and it's very real. You open your laptop, the coffee is still too hot to drink, and the build is red. You click in. The failing test is the one about the newsletter signup modal, which nobody has touched in months. You rerun it. It passes. You merge, slightly less sure of everything.

That's a flaky test: one that sometimes fails without anything being wrong. Each one feels like a small annoyance. Together they're one of the most expensive problems in testing.

One percent sounds fine

Imagine a gate of 150 automated tests that must all pass before a deploy goes out. Each test is pretty reliable. It flakes, failing for no real reason, just 1% of the time.

What fraction of perfectly good deploys get blocked?

For a deploy to pass, all 150 tests have to pass. The chance of that is 0.99 multiplied by itself 150 times, which is about 0.22. So roughly 78% of deploys get blocked by at least one flake, even when the code is fine.

Read that again. Every individual test is 99% reliable, and the gate is wrong more than three times out of four.

That's how teams end up with a culture of "just rerun it." Nobody decided to ignore red builds. The math decided for them.

What one retry does

Now give each failing test one automatic retry. A test only counts as failed if it fails twice in a row. If flakes are independent, the chance of a flaky test failing twice is 1% of 1%, or 0.01%.

Across 150 tests, the chance of a false block drops to about 1.5%.

From 78% to 1.5%, with one retry. That's a big enough difference that we treat retry policy as a design decision, not a CI setting someone tweaks when they're annoyed.

There's a catch, and it matters. Retries hide flakiness. If a test needs its retry every other run, you want to know about it, because it's either telling you about a real race condition or it's about to become useless. So the retry has to be recorded, and a test that leans on it too often should lose its place in the gate.

Flaky tests don't start in the gate

That last point shapes how we're designing Testomatica. A new test, whether a person wrote it or it was generated from a finding, doesn't get to block a release on day one. It starts in quarantine. It runs, its results are recorded, and it can't stop anything.

Only after a long stable streak, across several commits and several days, does it get promoted to the blocking gate. We're planning to use a sequential test (Abraham Wald's SPRT) to decide when the streak is long enough, which in practice works out to around thirty clean observations. It's slower than shipping every new test straight into the gate. It's also how you keep the gate worth listening to.

Not every red is a test failure

One more source of blue Mondays: the test didn't fail, the environment did. Staging was mid-deploy. A DNS lookup timed out. The test database was being restored. These show up as red tests, and people learn to shrug at them.

We want the system to tell those apart. An infrastructure failure should be reported as an infrastructure failure, and retried or skipped, not counted against the test and not shown to you as a bug in your code.

Google's testing blog wrote about the cost of flaky tests back in 2016, and the advice then was the same as now: measure flakiness, isolate it, don't let it erode trust. We just want a system that does that by default, so nobody has to.

If you're staring at a red build this morning, check whether it's red twice before you believe it. And maybe let the coffee cool first.