Muller - The Tyranny of Metrics

THE TYRANNY OF METRICS

THE
TYRANNY
OF
METRICS

JERRY Z. MULLER

PRINCETON UNIVERSITY PRESS
PRINCETON & OXFORD

Copyright 2018 by Princeton University Press

Published by Princeton University Press,

41 William Street, Princeton, New Jersey 08540

In the United Kingdom: Princeton University Press,

6 Oxford Street, Woodstock, Oxfordshire OX20 1TR

press.princeton.edu

Jacket design by Chris Ferrante

Book epigraph from Everything: A Book of Aphorisms, 2nd ed., by Aaron Haspel. Copyright 2015 by Aaron Haspel. Published by Good Books and reprinted here by permission of the author.

All Rights Reserved

ISBN 978-0-691-17495-2

British Library Cataloging-in-Publication Data is available

This book has been composed in Sabon Next, Alternate Gothic, and Gotham

Printed on acid-free paper.

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

FOR PETER J. DOUGHERTY

Those who believe that what you cannot quantify does not exist also believe that what you can quantify, does.

AARON HASPEL

CONTENTS

THE TYRANNY OF METRICS

INTRODUCTION

Based on the real-life experiences of its creators, David Simon and Ed Burns, the HBO series The Wire is regarded by some as among the greatest cultural documents of our age. And with good reason. Focused on a single American city, Baltimore, the series drills down into a few major institutionsthe police, the school system, municipal politics, the pressand provides an X-raylike image of their workings and dysfunctions. The series has attracted an international audience because its themes of organizational dysfunction resonate broadly across Western societies.

One of the recurrent themes of The Wire is the salience of metrics: of measured performance as the hallmark of accountability. Police commanders are obsessed with hitting the numbersfor example, cases solved, drug arrests, crime ratesand they do so by a variety of means that sacrifice effectiveness to meeting statistical targets. Politicians demand numbers that attest to police success in controlling crime. So the police units do their best to avoid having murders attributed to their district: when it turns out that a drug gang has been disposing of bodies in abandoned houses, the homicide sergeant discourages their discovery, since that would diminish the clearance rate, the metric of the percentage of crimes solved. Much of the plot revolves around dedicated detectives seeking to develop a complex criminal case against a major drug lord. But since building that case will take months if not years, they are discouraged from doing so by the higher-ups, who want the cops to rack up favorable metrics by arresting lots of low-level drug dealers, despite the fact that those arrested will be replaced almost instantly. The mayors office demands that the rate of major crimes decline by 5 percent before the end of the year, a target that can be reached only by overlooking actual crimes or downgrading their seriousness. In each case, they are engaged in juking the statsimproving their metrics either by distorting actual results, or by diverting their time and effort from crime prevention to less productive work.

Another plot line involves an ex-cop who teaches in a middle school in a neighborhood plagued by poverty, drug abuse, and family fragmentation. Students in the school perform poorly, and the school is in danger of being closed if the test scores of its students do not improve. So, in the six weeks before the standardized English reading and writing tests are to be administered, the teachers are instructed by their principal to focus all of class time on practicing for the tests, ignoring other subjects entirely (a strategy euphemistically referred to as curriculum alignment). Teaching to the test, like juking the stats, is a way in which institutions are perverted, as effort is diverted from the institutions true purpose (education) to meeting the metric targets on which its survival has come to depend.

The distortive effects of performance metrics are felt at least as much across the Atlantic, in Great Britain.measured performance. The cost of this tactic is that patients at greater risk for a failed surgery are left to an almost certain death without surgery.

Bodies is a medical drama, but the phenomena it depicts exist in the real world. Numerous studies have shown that when surgeons, for example, are rated or remunerated according to their success rates, some respond by refusing to operate on patients with more complex or critical conditions. Excluding the more difficult casesthose that involve the likelihood of poorer outcomesimproves the surgeons success rates, and hence their metrics, their reputation, and their remuneration. That of course comes at the expense of the excluded patients, who pay with their lives. But those deaths do not show up in the metrics.

As well see, gaming the metrics occurs in every realm: in policing; in primary, secondary, and higher education; in medicine; in nonprofit organizations; and, of course, in business. And gaming is only one class of problems that inevitably arise when using performance metrics as the basis of reward or sanction. There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledgeknowledge that seems solid but is actually deceptive.

We live in the age of measured accountability, of reward for measured performance, and belief in the virtues of publicizing those metrics through transparency. But the identification of accountability with metrics and with transparency is deceptive. Accountability ought to mean being held responsible for ones actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement, as if only that which can be counted really counts. Another assumption that is often taken for granted is that accountability demands that measurement of performance be made public, that is, transparent.

The metric fixation is the seemingly irresistible pressure to measure performance, to publicize it, and to reward it, often in the face of evidence that this just doesnt work very well.

Used properly, measurement, as well see, can be a good thing. So can transparency. But they can also distort, divert, displace, distract, and discourage. While we are bound to live in an age of measurement, we live in an age of mismeasurement, over-measurement, misleading measurement, and counter-productive measurement. This book is not about the evils of measuring. It is about the unintended negative consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurementnot metrics, but metric fixation.

We are often told that gathering metrics of measured performance and then making them available to the public is a way to improve the functioning of our institutions. Nowhere have the virtues of accountability, performance metrics, and transparency been more touted than in the field of medicine. And understandably so, for nowhere are the stakes higher. The health sector not only makes up over 17 percent of the U.S. economy, but lives are also on the line. Surely, the logic goes, measures of performance can help save dollars and save lives.

Gathering standardized information about the success rates of surgeons, or the survival rate of patients admitted to particular hospitals, is supposed to be helpful. For if doctors or hospitals are remunerated by government agencies or private insurers based on their success rates in keeping patients alive, then such measurements should create incentives for better care. And if the success rates of doctors and hospitals are publicized, the resulting transparency will allow the public to choose among doctors and among hospitals. All in all, metrics, accountability, and transparency will provide the cure for what ails the medical professions. What could go wrong?