David H. Freedman - Wrong: Why experts* keep failing us--and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more

Also by David H. Freedman

A Perfect Mess: The Hidden Benefits of DisorderHow Crammed
Closets, Cluttered Offices, and On-the-Fly Planning Make the World
a Better Place (with Eric Abrahamson, 2007)

Corps Business: The 30 Management Principles of the
U.S. Marines (2000)

At Large: The Strange Case of the Worlds Biggest Internet Invasion
(with Charles C. Mann, 1998)

Brainmakers: How Scientists Are Moving Beyond Computers to
Create a Rival to the Human Brain (1995)

Copyright 2010 by David H. Freedman

All rights reserved. Except as permitted under the U.S. Copyright Act of 1976,
no part of this publication may be reproduced, distributed, or transmitted in any
form or by any means, or stored in a database or retrieval system, without the
prior written permission of the publisher.

Little, Brown and Company
Hachette Book Group
237 Park Avenue, New York, NY 10017
www.hachettebookgroup.com

First Edition: June 2010

Little, Brown and Company is a division of Hachette Book Group, Inc.
The Little, Brown name and logo are trademarks of Hachette Book Group, Inc.

C ONTENTS

If he is weak in the knees, let him not call the hill steep.

H ENRY D AVID T HOREAU

Success consists of going from failure to failure without
loss of enthusiasm.

W INSTON C HURCHILL

I m sitting in a coffee shop in a pediatric hospital in Boston, hard by a nine-foot-tall bronze teddy bear, with a man who is going to perform a surprising trick. Im thinking of an article recently published in a prestigious medical journal, an article that reports the results of a research study, and he will tell me whether or not the study is likely to turn out to be right or wrong. Its the sort of study that your doctor might read about, and that you might learn about from a newspaper, website, or morning TV news show. It may well be that the results of this study will change your lifethey might convince you to start eating or avoiding certain foods to lower your risk of heart disease, or to take a certain drug to help you beat cancer, or to learn whether or not you are carrying a gene linked to vulnerability to a mental illness. But this man wont need to hear any of the particulars of the study to perform his feat. All he needs to know is that it was a study published in a top journal.

His prediction: its wrong. Its a prediction that strikes at the foundation of expertise and our trust in it.

The man is John Ioannidis, a doctor and researcher whose specialty is calculating the chances that studies results are false. For someone dedicated to spotlighting the inadequacies of his colleagues lifework, Ioannidis is pleasant, polite, and soft-spoken, even if he discreetly radiates the fidgety energy of someone who habitually packs too much into his day. He looks young for a man heading into his midforties, with a slight build, a wavy mop of fine, dark hair, and a thin mustache. Also a bit surprising about Ioannidis is that he is highly regarded by his peers. Communities usually find ways to marginalize those who expose their flaws, but the world of medical research, in which extraordinary talent and effort are prerequisites for attaining even the lowest rungs of recognition, has kept Ioannidis in demand via the fields standard trappings of success: prestigious appointments, including one at the world-class TuftsNew England Medical Center and another at the University of Ioannina Medical School in his native Greece; frequent citations by colleagues of his work, some of which has been published in the fields top journals; and a stream of invitations to speak at conferences, where he is generally a big draw.

Theres no standard career path to becoming a deconstructor of wrongness, and Ioannidis took a roundabout route to it. Born in 1965 in the United States to parents who were both physicians, he was raised in Athens, where he showed unusual aptitude in mathematics and snagged Greeces top student math prize. By the end of college, he seemed on track for a career as a mathematician. But he had come to feel the family pull of medicine and, not wanting to turn his back on math, decided to combine the two and become a medical mathematician. I didnt know exactly what such a thing might be, he says, but I felt sure there was some important component of medicine that was mathematical. He graduated first in his class at the University of Athens Medical School, then shipped off to Harvard for his residency in internal medicine, followed by a research and clinical appointment at Tufts in infectious diseases. The math had to this point remained in the background, but in 1993, while at Tufts, he saw his chance to even things up a bit. There was growing interest in the new field of evidence-based medicinethat is, trying to equip physicians to do not merely what they had been taught to assume would help patients but what had been rigorously proven in studies would help patients. Amazingly, most medical treatment simply isnt backed up by good, quantitative evidence, says Ioannidisnews that would likely come as a surprise to most patients. Distilling this sort of knowledge out of a chaos of patient data often requires more statistical-analysis firepower than clinical researchers bear, providing an opening for Ioannidis to make a mark.

Carrying his new interest to joint appointments at the National Institutes of Health and Johns Hopkins in the mid-1990s, Ioannidis began to look for interesting patterns in those medical-journal studies that explore how patients fare with certain treatments. Such studies are essentially the coin of the realm when it comes to communicating solid evidence of treatment effectiveness to physicians. A good doctor, it is presumed, scans the journals for the results of these studies to see what works and what doesnt on which patients, and how well and with what risks, modifying her practices accordingly. Does it make sense to prescribe an antibiotic to a child with an ear infection? Should middle-aged men with no signs of heart disease be told to take a small, daily dose of aspirin? Do the potential benefits of a particular surgical intervention outweigh the risks? Studies presumably provide the answers. In examining hundreds of these studies, Ioannidis did indeed spot a patterna disturbing one. When a study was published, often it was only a matter of months, and at most a few years, before other studies came out to either fully refute the findings or declare that the results were exaggerated in the sense that later papers revealed significantly lesser benefits to the treatment under study. Results that held up were outweighed two-to-one by results destined to be labeled never mind.

What was going on here? The whole point of carrying out a study was to rigorously examine a question using tools and techniques that would yield solid data, allowing a careful and conclusive analysis that would replace the conjecture, assumptions, and sloppy assessments that had preceded it. The data were supposed to be the path to truth. And yet these studies, and most types of studies Ioannidis looked at, were far more often than not driving to wrong answers. They exhibited the sort of wrongness rate you would associate more with fad-diet tips, celebrity gossip, or political punditry than with state-of-the-art medical research.

The two-out-of-three wrongness rate Ioannidis found is worse than it sounds. He had been examining only the less than one-tenth of one percent of published medical research that makes it to the most prestigious medical journals. In other words, in determining that two-thirds of published medical research is wrong, Ioannidis is offering what can easily be seen as an extremely optimistic assessment. Throw in the presumably less careful work from lesser journals, and take into account the way the results end up being spun and misinterpreted by university and industrial PR departments and by journalists, and its clear that whatever it was about expert wrongness that Ioannidis had stumbled on in these journals, the wrongness rate would only worsen from there.