Kaiser Fung - Numbers Rule Your World: The Hidden Influence of Probabilities and Statistics on Everything You Do

NUMBERS RULE YOUR WORLD

THE HIDDEN INFLUENCE OF PROBABILITY AND STATISTICS ON EVERYTHING YOU DO

KAISER FUNG

Copyright 2010 by Kaiser Fung. All rights reserved. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher.

ISBN: 978-0-07-174541-3

MHID: 0-07-174541-6

The material in this eBook also appears in the print version of this title: ISBN: 978-0-07-162653-8, MHID: 0-07-162653-0.

All trademarks are trademarks of their respective owners. Rather than put a trademark symbol after every occurrence of a trademarked name, we use names in an editorial fashion only, and to the benefit of the trademark owner, with no intention of infringement of the trademark. Where such designations appear in this book, they have been printed with initial caps.

McGraw-Hill eBooks are available at special quantity discounts to use as premiums and sales promotions, or for use in corporate training programs. To contact a representative please e-mail us at bulksales@mcgraw-hill.com.

TERMS OF USE

This is a copyrighted work and The McGraw-Hill Companies, Inc. (McGraw-Hill) and its licensors reserve all rights in and to the work. Use of this work is subject to these terms. Except as permitted under the Copyright Act of 1976 and the right to store and retrieve one copy of the work, you may not decompile, disassemble, reverse engineer, reproduce, modify, create derivative works based upon, transmit, distribute, disseminate, sell, publish or sublicense the work or any part of it without McGraw-Hills prior consent. You may use the work for your own noncommercial and personal use; any other use of the work is strictly prohibited. Your right to use the work may be terminated if you fail to comply with these terms.

THE WORK IS PROVIDED AS IS. McGRAW-HILL AND ITS LICENSORS MAKE NO GUARANTEES OR WARRANTIES AS TO THE ACCURACY, ADEQUACY OR COMPLETENESS OF OR RESULTS TO BE OBTAINED FROM USING THE WORK, INCLUDING ANY INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA HYPERLINK OR OTHERWISE, AND EXPRESSLY DISCLAIM ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. McGraw-Hill and its licensors do not warrant or guarantee that the functions contained in the work will meet your requirements or that its operation will be uninterrupted or error free. Neither McGraw-Hill nor its licensors shall be liable to you or anyone else for any inaccuracy, error or omission, regardless of cause, in the work or for any damages resulting therefrom. McGraw-Hill has no responsibility for the content of any information accessed through the work. Under no circumstances shall McGraw-Hill and/or its licensors be liable for any indirect, incidental, special, punitive, consequential or similar damages that result from the use of or inability to use the work, even if any of them has been advised of the possibility of such damages. This limitation of liability shall apply to any claim or cause whatsoever whether such claim or cause arises in contract, tort or otherwise.

To
Mum, Dad, Grandma, and Evelyn

Contents

Acknowledgments

I would like to acknowledge the guidance and assistance of Grace Freedson, Michele Paige, Micah Burch, Kate Johnson, Steven Tuntono, Beth McFadden, Talbot Katz, and my editors, John Aherne and Joseph Berkowitz. My two sisters and brother made invaluable contributions as my most plain-spoken critics.

In addition, throughout this project, I was inspired by fans of my Junk Charts blog, www.junkcharts.typepad.com .

Introduction

This is not another book about damned lies and statistics. That evergreen topic has inspired masterworks from Darrell Huff, John Allen Paulos, Ed Tufte, and Howard Wainer, among others. From the manipulative politician to the blundering analyst, from the amateur economist to the hard-selling advertiser, we have endless examples of what can go wrong when numbers are misused. Cherry-picking, oversimplifying, obfuscatingwe have seen them all. This book takes a different direction, a positive position: I am interested in what happens when things go right, which is to say, what happens when numbers dont lie.

The More We Know We Dont Know

What will we learn from Bernie Madoff, the New Yorkbased fund managerswindler who impoverished an exclusive club of well-to-do patrons over three decades until he confessed in 2008? Or from the Enron executives whose make-believe accounting wiped out the retirement savings of thousands of employees? Perhaps we ought to know why the reams of financial data, printed statements, and official filings yielded few clues to the investors, auditors, and regulators who fell for the deception.

What will we learn from the Vioxx debacle in which the Food and Drug Administration conceded, five years after blessing its initial release, that the drug had caused ten thousand heart attacks? Perhaps we ought to know why widely available health and medical information and greater scale and sophistication of clinical trials did not spare Vioxx inventor Merck, doctors, or patients from overlooking the deadly side effects.

We ought also to ask why, despite having access to torrents of stock data and company reports, most of us have not made a killing in the stock market. Despite tallying up the nutritional information of every can and every packet of food, most of us have not achieved the hoped-for bodily downsizing. Despite heavy investment in information technology, flight delays and traffic jams continue to get worse. Despite detailed records of our shopping behavior, many companies have but the slightest clue when we call their service centers. Despite failing to arrest cancer in patients during large-scale clinical trials, beta-carotene and vitamin pills keep flying off the pharmacy shelves.

These examples reveal the unpleasant surprise that the modern obsession with measurement has made us none the wiser. We collect, store, process, and analyze more information than ever beforebut to what end? Aristotles wisdom has never been more relevant than it is today: the more we know, the more we know we dont know.

Stories of a Positive Nature

We begin to overcome these failures by examining positive examples of how enterprising people are making sensible use of the new information to better our world. In the next five chapters, you will meet engineers who keep the traffic flowing on Minnesota highways, disease detectives who warn us about unsafe foods, actuaries who calculate how much Floridians must pay to insure homes against hurricanes, educators who strive to make standardized tests like the SAT fair, lab technicians who scrutinize blood samples from elite athletes, data miners who think they can detect our lies, lottery operators who face evidence of fraud, Walt Disney scientists who devise ever-clever ways to shorten queues, mathematicians whose ideas have set off the explosion of consumer credit, and researchers who offer the best tips for air travel.

These ten portraits feature some special men and women whose work is rarely celebrated openly. The reason for this neglect is that their achievement is not of invention, for which we shower awards and accolades, but of adaptation, of refinement, of salesmanship, and of perseverance. Their expertise is applied science.

The Statistical Way of Thinking