The coverage universe: mapping everything worth testing

Ask a team "how well tested is this product?" and you'll usually get one of two answers. A number, like "82% line coverage." Or a shrug.

The number is real, but it answers a narrow question. It tells you which lines of code executed while tests ran. It doesn't tell you whether the product page is readable with a screen reader, whether the checkout survives a flaky 3G connection, or whether the search results page has a sensible title for Google. Those things aren't lines of code. They're properties of a product.

The shrug is more honest. It just isn't very useful.

We wanted something in between: a list of everything worth testing, where each entry has a clear owner and a clear status. We've been calling it the coverage universe, partly as a joke about how big it gets.

Four axes

Every item on the map sits somewhere along four axes.

Level. Is this a unit of logic, an API, a page, or a whole user journey? "Coupon math is correct" is a different item from "a shopper can apply a coupon and check out."

Surface. Where does it live? The storefront, the account area, the admin panel, emails, the public API. Some products have surfaces their teams forget they have, like a sitemap or the structured data that search engines read.

Quality dimension. What kind of "working" are we talking about? We use nine: function, data, security, performance, accessibility, SEO, visual, device and usability. A product page can be functionally perfect and still fail four of the other eight.

Lifecycle. When does this matter? On every deploy, before a release, after a migration, once a quarter? A broken image matters on every deploy. A check that the cookie consent text matches the current legal copy probably doesn't.

Put together, one item might read: journey level, storefront, accessibility, every release: a keyboard-only user can go from search to order confirmation. That's specific enough to own and specific enough to tell whether it's covered.

Who covers it

Listing items is the easy half. The harder half is being honest about who's actually checking each one.

We use three executors:

1. Code and tools. Automated tests, rule checks, scanners. These are the only things allowed to block a release on their own, because they give the same answer every time.
2. Agents. Explorers that roam with a persona and a charter. They find things and propose them. They never block a release alone.
3. Humans. Judgment calls, things that need taste or context, and the final sign-off.

Every item gets assigned to one or more of these. And there's a direction to it: whatever humans or agents find, we try to turn into code. If a person notices the order confirmation email has a broken logo, the next step is a check that looks for broken images in emails, so nobody has to notice it again.

The goal isn't for everything to end up as code. Some things never should. "Does this onboarding flow feel confusing?" is a human question. The goal is that nothing on the map is assigned to nobody.

Why bother with all this

Mostly because it makes gaps visible.

Without the map, an untested area is invisible. It doesn't show up in a coverage report because there's no code to count, and it doesn't show up in a test plan because nobody wrote it down. With the map, it shows up as an item with no executor next to it. That's an uncomfortable line to look at, which is the point.

It also changes the "how well tested is this?" conversation. Instead of one number, you can say: these items are guarded by code on every deploy, these are explored by agents, these are on a human's list for the next release, and these few are unassigned and we know it.

Where it's still rough

We'll be honest about a few things.

The map can get big fast. Multiply surfaces by dimensions by levels and you get far more cells than anyone wants to maintain. Most cells are empty on purpose. We don't need a performance item for the privacy policy page. Deciding which cells deserve an entry is still a judgment call, and we're writing down the reasoning as we go.

Items drift. A product changes and the map has to keep up. Our current plan is to have the planner flag pages and endpoints it sees in a diff that don't belong to any item yet. We'll see how noisy that is.

And ordering within the map is its own topic. We use risk to decide what to look at first, but never to decide what to drop. We wrote about that in Risk scores are for sorting, not for budgeting.

If you were drawing this map for your own product today, which of the nine dimensions would have the emptiest column? Our guess is that it won't be function.