Early on, I reported QA metrics the way everyone seems to at first: bugs filed this sprint, test cases executed, percentage passed. The numbers looked rigorous and said almost nothing. QA metrics are only useful when they measure outcomes — did users hit problems, how fast did we notice, can we trust our own signals — and most of the popular ones measure activity instead.
The metrics that lie
Bug counts are the classic. A high count can mean a buggy build, a thorough tester, a generous definition of "bug," or a week spent on a crumbly legacy screen. A low count can mean quality or can mean nobody looked. Worse, the moment bug counts become a target, they get gamed — trivial issues get split into three tickets, and developers and QA start negotiating severity instead of fixing things.
Test-case counts lie the same way. "We have 800 test cases" says nothing about whether they cover what's risky, whether they're duplicates, or whether half are stale steps for screens that no longer exist. Raw pass percentage joins them: 98% passing sounds great until you learn the failing 2% is checkout.
QA metrics that matter
- Escaped defects. Bugs found in production that testing should reasonably have caught. This is the closest thing QA has to a ground-truth quality measure. The count matters less than the review: for each escape, ask which test, environment, or data set was missing. Escapes drive better decisions than any dashboard I've built.
- Time-to-detect. How long a bug existed before anyone noticed — roughly, when it shipped versus when it was found. Shrinking it is the whole argument for automation in CI: once our Maestro smoke suite ran on every pull request, whole classes of bugs went from "found next sprint" to "found before merge," and our regression pass takes roughly half the time it used to.
- Flake rate. The share of test failures that were the test's fault, not the product's. This is the trust metric. When flake rate climbs, people stop reading red builds, and an automation suite nobody believes is worse than none. I track which tests fail-then-pass on retry and treat repeat offenders as bugs in the suite.
- Coverage against risk, not against numbers. Instead of "how many test cases," ask: of the flows that would cost us most if broken — payments, login, data loss — which have automated checks, which have manual passes, which have nothing? A short table of risky areas versus coverage status is more honest than any percentage.
Reporting trends to stakeholders
A single number is a fact; a trend is a story, and stakeholders act on stories. "Twelve bugs this sprint" invites the useless question "is that a lot?" — "escaped defects have fallen for three consecutive releases since we added the payment regression suite" invites investment. Three habits that made my reporting land better:
- Report the same three or four metrics every time, so changes are visible against a baseline.
- Pair every chart with one sentence of interpretation. Stakeholders don't need the data; they need what it means and what you'll do about it.
- Never report a metric you wouldn't want optimized. Whatever you present becomes a target, so present things that are good even when gamed — nobody games escaped defects downward except by shipping better software.
If you can track only one thing, track escaped defects and hold a ten-minute review for each. It's the metric that most directly asks "is QA working?" — and the review generates your roadmap for free.
Metrics don't improve quality; they aim attention. Aim it at what users experience, how fast you notice, and whether your signals deserve trust — and let the bug-count charts retire quietly.