The ineffectiveness of measurement in a bureaucracy
Corporations that are dissatisfied with their progress at innovating, often set up measurement systems to assess the current status and guide future progress. While measurement can play a role in managing innovation, measurement often ends up serving purposes that have nothing to do with the truth. The following 21 lessons on the uses, abuses, and half-truths of the measuring innovation can help save firms from making simple mistakes.
General Principles Of Measurement
1. Measurement Is Ubiquitous In Society And Takes Many Forms
Measurement includes quantitative methods and non-quantitative approaches (e.g. jury trials). It includes permanent measurement systems (e.g. GDP) and ad hoc systems designed to resolve specific issues. Measurement systems may be used to implement agreed ways of looking at things (e.g. GDP) even though they are known to be imperfect. In other cases, measurement may be aimed at building agreement.
2. It Is Unclear To What Extent Measurement Works
According to one source, there is little public evidence of the contributions that performance measurement has made to improve decision making. In part, that is because “not everything that counts can be counted, and not everything that can be counted counts” (falsely attributed to Einstein) It is also in part because the balance of successes and failures is not known. Although some benefits have been immense, some costs have also been colossal. It is also in part because decision-makers don't always act on the findings of measurement.
3. Performance Measures Have Many Uses
Measurements can be used to do retrospective assessments of realized, observed, and measured impacts. Did it work? Did it produce the desired results?
Second, performance measures can be used to assess the best direction in which to head.
Finally, measures can benchmark accomplishments against historical or international measures and advocate for actions.
4. Measures Have Little Meaning In The Abstract
The promise and limitations of measuring innovation depend on the decisions that are made. Measurements are necessary and productive for certain types of decisions, problematic for others, and harmful for still others. A critical issue is who is asking the question and why.
5. Measures Can Provide Useful Baselines For Assessing Accountability.
Performance measures can help executives avoid “fads” that direct attention in unproductive ways. Measurements can document how some activities do not have a solid evidentiary base and signal that it is time to move on.
6. The Returns On Measurement Of Innovation Can Be Uncertain, Long Term, And Circuitous
This insight makes it difficult to put innovation in a strict accountability regime. In addition, impacts typically depend on complementary actions by other entities.
7. The Benefits From Failure Are Often Underestimated
Failure can redirect research into extremely productive directions.
8. You Can’t Manage What You Can’t Measure
There is some truth to this saying which is falsely attributed to Peter Drucker. Measurement is not always possible. Some aspects of truth are not amenable to management. Sometimes there is no time to measure.
9. When A Measure Becomes A Goal, It Ceases To Be A Measure (Goodhart’s Law)
This appears to have happened in part to Fred Reichheld’s highly successful NPS measure of customer satisfaction.
10. Measurement Sometimes Becomes A Fetish
Our current culture is crazy about numbers. Sometimes numbers are pursued for their own sake, like wearing a fashionable new suit.
11. The Act Of Measuring Can Change What Is Measured
This is true in both quantum physics and in human affairs.
The Value of Measuring Innovation
These general principles lead to the following practical applications in the measurement of innovation.
12. Measurement As A Tool For Discovering The Truth About Innovation
Three elements are necessary for effective measuring systems in a firm:
· Measures must be external to the firm; (measures internal to the firm are often not worth much)
· Measures must be of actual customer behavior, not surveys of customer opinion (customers themselves often don't know what they need)
· The firm must find a way to bring the numbers to life; Amazon and Microsoft accomplish this in different ways: Amazon by “future press releases”; and Microsoft by way of CEO Nadella's personal passion and obsession with customer usage.
Caveat: For measures of innovation to be effective, decision-makers must want to know the truth, even if it contrary to their own views or interests.
13. Measurement As A Tool For Learning
The success of the goal of demystification depends on the level of curiosity of those conducting the measurement.
14. Measurement As A Tool For Fighting Hubris
This only works if decision makers are willing to discuss the undiscussable, namely, that some of their deeply held beliefs may have no evidentiary basis.
Abuses Of Measurement In Corporations
15. Measurement As A Tool For Supporting A Cause
Launching and implementing innovation can be difficult in a large organization, with opposition coming from many directions. In defense, those leading innovation efforts may choose to report on measures that emphasize success and de-emphasize measures that point to problems. This abuse of truth may be justified as necessary to win the innovation war: as in any war, truth is the first casualty. The suppression of truth may be done deliberately or unconsciously.
16. Measurement As A Goal In Itself
Measurement for measurement’s sake is common in parts of academia and also the evaluation departments of large corporations. The measurement is sometimes done to advance the cause of those doing the measuring than to learn anything in particular.
17. Measurement As A Tool For Winning Hearts And Minds To Support Innovation
Measurement is usually not very good for winning either hearts or minds. The Confirmation Bias is a powerful hindrance to perceiving truth by those who have settled negative views about the worth of pursuing the innovation. Leadership storytelling can be more effective for winning hears than dry numbers. Sometimes retirement or death is the only solution to removing the resistance to necessary innovation.
18. Measurement As A Tool For Exercising Power
Those in power may decide use only those measures that support their cause. They may select measures that highlight the strengths of their position and adjust the measures that reach “the wrong result.” This happens in corporations as well as governments.
19. Measurement As A Tool For Defeating Truth
Before Francis Bacon, science consisted of experiments that were done to confirm what was already known and maintain the status quo. Modern science began, and began to advance, only when it was used to disprove what was thought to be true. Measurement in modern corporations often resembles pre-Baconian science.
20. Measurement As A Tool For Crushing Opposition
In corporations, measuring tools can be used by executives who want to preserve the status quo. In government, too, the GDP omits large parts of the digital economy but there is little effort at improvement.
21. Measurement As A Tool For Self-Delusion
The level of self-delusion with a measure can increase with the hierarchical level of the perceiver. It can be difficult to see the ground from the top of a pyramid.
Conclusion
Measurement can play a hugely useful role in managing innovation when people truly want to know the answer to a question, they design a measurement system correctly and they are ready to act promptly on what they learn. In many cases, none of these conditions are met and so vast amounts of money are spent on measurement systems that are mostly spinning wheels. Firms can enhance their chances of success by understanding the human factors they are dealing with.
And read also:
