Ninety years of warnings, six KPIs of ignoring them

"It is wrong to suppose that if you can't measure it, you can't manage it — a costly myth." Deming, The New Economics (1993):

At the Institute for Marine Mammal Studies in Mississippi, the dolphins were trained to keep their own pool clean. Any litter that fell into the water, a dolphin would hold until a trainer came by, then trade it for a fish. A sensible program: reward the behavior you want, get more of it.

A dolphin named Kelly noticed something the trainers had never specified: the fish was paid per item, not per amount. So she began hiding paper under a rock at the bottom of the pool. When a trainer passed, she would swim down, tear off a small piece, and trade it. Then another piece. One soggy sheet of paper became a steady stream of fish.

Then one day a gull flew into her pool. Kelly caught it, held it, "traded it in" and received a large fish reward, because a gull is considerably more impressive than a scrap of paper. So Kelly took her next fish and hid it under the rock. When no trainers were watching, she used it to bait gulls. She caught the gulls and traded them in too. Then she taught the trick to her calf.

The Guardian, "Why dolphins are deep thinkers," 3 July 2003

Nobody at the institute intended to fund a paper-shredding operation or a gull-baiting enterprise. They set a measurable target, attached a reward, and a brain that isn't even human optimized exactly what was rewarded, not what was best.

We have known this for ninety years

Humans have been documenting this failure mode, with dates and citations, for nearly a century. In 1936, Robert K. Merton published "The Unanticipated Consequences of Purposive Social Action" in the American Sociological Review — a systematic account of how deliberate interventions produce outcomes nobody intended. In 1975, the economist Charles Goodhart, writing about monetary policy, put it in one sentence: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." (The version we all quote — "When a measure becomes a target, it ceases to be a good measure" is Marilyn Strathern's generalization, from 1997.) In 1979, Donald T. Campbell stated it as a law of his own: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor".

This wasn't confined to academia. In 1986, W. Edwards Deming, the man postwar Japanese industry credits for its quality revolution, made the elimination of numerical quotas and management by numbers one of his fourteen points for management in Out of the Crisis. And in 1996, Robert D. Austin published Measuring and Managing Performance in Organizations, an entire book applying this literature to one industry in particular: software. This thesis has been settled, in science and in management, for ninety years.

And yet, this year

Fast forward ninety years. It is 2026, and I still see articles advocating per-tester KPIs and advising how to set them up, complete with formulas, thresholds, and maturity-graded targets. The genre is thriving. Every quote below is verbatim, from some recent examples; I am not saying which, because it does not matter: they are interchangeable.

These articles often start with an authority: "The often-heard quote of W. Edwards Deming, a quality-control leader, aptly captures the importance of KPIs: 'We cannot control things which we cannot measure.'"

Deming never said it. The W. Edwards Deming Institute maintains a page debunking this exact family of quotes, which keep getting stapled to his name because they sound like something a quality guru ought to have said. This is the same Deming who, as you just read, put the elimination of management by numbers into his fourteen points for management. Invoking him to sell KPI quotas is like quoting a fire marshal in praise of arson.

Here are some KPIs these articles advise, each with a question: what will a tester do to hit this number?

1. Requirements coverage. "Target: 95%+ for mature teams." The metric counts test cases attached to requirements, not risk covered. The rational move is to write one trivial test case per requirement. One hundred percent coverage of shallow checks outscores eighty percent coverage of the paths that actually take your product down. The dashboard turns green; the risk stays exactly where it was.

2. Defects fixed per day. *"Formula: Defects fixed in period/ Number of days\100. Typical target: 3-7 per day for balanced teams." First, testers do not fix defects; developers do, so this KPI for testers measures someone else's work. Second, multiplying a daily rate by 100 means the formula computes nothing: the result is not a percentage of anything. Third, as a quota, it rewards exactly two behaviors: filing trivia that is easy to fix, and splitting one bug into five tickets. Reminds me of Kelly the dolphin tearing the sheet of paper into scraps. Same reward schedule, same optimization.

A genuine question: what should a tester actually do differently to increase this number from 2 to 5 without gaming the system?

3. Test execution speed. The number of tests executed per day or week. The rational move: run the fastest, shallowest tests, and run them often. Under this metric, exploratory testing — usually one of the highest-value activity a skilled human tester performs — scores exactly zero. Activities that score zero under a KPI do not survive contact with a performance review. So it stops happening.

4. Automation percentage. "Should be 30-50% for startups and 60-85% for enterprises." Automate whatever is cheapest to automate, regardless of value — the ratio cannot tell the difference. And there is a shortcut: deleting manual test cases raises the percentage without automating anything. The metric improves as the safety net shrinks.

5. Active defects. "Best practice: Set a threshold like <20 for smaller projects." The most damaging of the six. A ceiling on open bugs converts every bug report into a liability. Testers hesitate to log; triage gets creative with "works as designed"; known issues migrate out of the tracker and into Slack threads and tribal memory. This is a machine for hiding information from yourself.

6. Severe bugs. A threshold on the number of severe defects. The bug does not get less severe. The label does. Once the count is capped, severity stops being an assessment and becomes a negotiation.

Many of these articles also suggest treating KPIs as a way to improve the work and the product, not as a way to blame individuals.

The typical target, remember, is three to seven defects fixed per day per tester.

The part where they are right

The impulse behind these articles is correct: wanting visibility into quality work is management doing its job. Testing has suffered for decades from being invisible until something ships broken, so the desire to measure is not a bad thing.

The mistake is how the measurement is used. Deming again, from The New Economics (1993): "It is wrong to suppose that if you can't measure it, you can't manage it — a costly myth." And in Out of the Crisis he approvingly quotes Lloyd Nelson: "the most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them."

Austin gave us the precise mechanism. There are two reasons to measure: informational (to understand and improve the system) and motivational (to steer people's behavior). The moment you take an informational measure and attach it to a person's evaluation, it inevitably becomes motivational, people begin managing the number instead of the work, and the measure stops informing. You don't just fail to motivate. You blind yourself.

What to do instead

The natural question at this point is: "then which metrics should we track instead?" It is the wrong question. There is no right universally applicable set. Quality is a match between what's desired and what's produced, it lives in people's experience of the product, and no ensemble of numbers converges on it, because checking whether the ensemble got it right would require the direct measurement whose impossibility made you build the ensemble. The numbers can tell you that something in the system changed: a trend broke, a point fell outside the natural variation. They cannot tell you the quality is good or bad. For that you do the unmeasurable work: read the bug reports, watch the users, talk to the testers, use the product. The most important figures really are unknown and unknowable, and management must take account of them anyway.

So the right question is not "which metrics?" but "why these metrics?", and the answer does not start with metrics at all, but with risks. Testing is an investment in reducing risk, not a scoreboard for people; my research on the economics of testing makes the full argument, but the short version is: you start from what actually threatens your product, derive the testing strategy from those prioritized risks, and only then, for that strategy, choose the few measures that tell you whether it is working. Every number you track exists because it answers a question you are asking about your own strategy. Nothing lands on a dashboard because an article said mature teams keep it above 95%.

Metrics chosen this way are temporary by design. Risks change, the strategy follows, so the set must be reviewed regularly: measures added, changed and removed if needed. A metric whose informational value has dropped to zero is not a harmless leftover on a dashboard; it is noise you are still paying attention to, and attention spent on noise ends in analysis paralysis: studying charts that no longer answer any question.

Measure the system, not the person. The measures that survive the review, you read as signals about the system, not verdicts on people. Signals to begin the analysis, never an appraisal. "Escaped defects went up this quarter, what has changed?" is a diagnostic. "Your escaped-defect number is too high" is an incentive to reclassify defects.

And whatever numbers you do keep: never attach them to an individual's evaluation. The moment you do, you are no longer observing your process, but rather measuring your people's creativity in defeating the observation. It took a dolphin just a few weeks to figure it all out.

Goodhart all the way down

One question remains: why does this genre keep appearing in 2026, ninety years into the literature?

Here is a hypothesis. "QA KPIs" is a high-volume search keyword. Content teams are measured on posts published, keywords ranked, organic sessions delivered. I do hope these articles are not written to change how anyone tests software (because if they are, the advice is way too dangerous to follow); I hope they are written to rank. Which makes them perfect specimens of their own subject: content optimized for a metric rather than for the reader, recommending metrics optimized for measurement rather than for quality. Goodhart's law all the way down.

So here is one last question, and I mean it without a gram of malice: are the people writing these articles measured on posts per month?

Because if they are, Because if they are, we have turned the internet into Kelly’s pool: each post earns a fish, and the water gets dirtier.