Jim Manzi - Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society

UNCONTROLLED

UNCONTROLLED

The Surprising Payoff of Trial-and-Error
for Business, Politics, and Society

JIM MANZI

A Member of the Perseus Books Group
New York

Copyright 2012 by Jim Manzi

Published by Basic Books,

A Member of the Perseus Books Group

All rights reserved. No part of this book may be reproduced in any manner whatsoever without written permission except in the case of brief quotations embodied in critical articles and reviews. For information, address the Perseus Books Group, 387 Park Avenue South, New York, NY 10016-8810.

Books published by Basic Books are available at special discounts for bulk purchases in the United States by corporations, institutions, and other organizations. For more information, please contact the Special Markets Department at the Perseus Books Group, 2300 Chestnut Street, Suite 200, Philadelphia, PA 19103, or call (800) 810-4145, ext. 5000, or e-mail .

Designed by Brent Wilcox

Library of Congress Cataloging-in-Publication Data

Manzi, Jim.

Uncontrolled : the surprising payoff of trial-and-error for business, politics, and society / Jim Manzi.

p. cm.

Includes bibliographical references and index.

ISBN 978-0-465-02931-0 (e-book)

1. Social sciencesExperiments. 2. Social sciencesResearch. 3. Experimental designSocial aspects. I. Title.

H62.M264 2012

001.4'34dc23

2012004409

10 9 8 7 6 5 4 3 2 1

For Margaret Jennings Manzi

As a young corporate strategy consultant, I once was on a team tasked with analyzing a proposed business program for a major retail chain. This company was considering a very large investment to improve its stores through a combination of a brighter layout, a different mix of merchandise, and more in-store employees to assist shoppers. The company believed consumers would positively receive this program, but the open question was whether it would lead to enough new sales to justify the substantial extra costs it would require. I developed a complicated analytical process to predict the size of the sales gain, including qualitative and quantitative consumer research, competitive benchmarking, and internal capability modeling. With great pride I described this plan to a partner in our consulting firm, who responded by saying, Okay... but why wouldnt you just do it to a few stores and see how it works?

This seemed so simple that I thought it couldnt be right. But as I began a series of objections to his question, I kept stopping myself midsentence. I realized that each of my potential responses was incorrect: an experiment really would provide the most definitive available answer to the question.

Over the next twenty years I became increasingly aware that real experiments were required for adjudicating among competing theories for the effects of business interventions intended to change consumer behavior. Cost changes often could be predicted reliably through engineering studies. But when it came to predicting how people would respond to interventions, I discovered that I could almost always use historical data, surveys, and other information to build competing analyses that would prove that almost any realistically proposed business program would succeed or fail, just by making tiny adjustments to analytical assumptions. And the more sophisticated the analysis, the more unavoidable this kind of subterranean model-tuning became. Even after executing some business program, debates about how much it really changed profit often would continue, because so many other things changed at the same time. Only controlled experiments could cut through the complexity and create a reliable foundation for predicting consumer response to proposed interventions.

This fundamental problem, albeit at vastly greater scale and severity, applies whenever we listen to impressive-sounding arguments that predict the society-wide effects of proposed major economic, welfare, educational, and other policy interventions. As an example, consider the deliberations around how to respond to the 2008 economic crisis. The country was facing a terrifying situation, and there was a widespread belief that emergency measures of some kind were called for as a matter of prudence. The incoming Obama administration proposed a large stimulus program, which led to an intense public debate in January and February 2009. Setting aside for a moment ideological predispositions and value judgments, this presented a specific technical issue: What would be the effects of any given stimulus proposal on general economic welfare? This was a practical question worth trillions of dollars that got to the reliability of our predictions about government programs.

The role of government spending and deficits in a major economic downturn has been the subject of extensive academic study for decades, and many leading economists actively participated in the public discussion in early 2009. Paul Krugman and Joseph Stiglitz, both Nobel laureates in economics, argued that stimulus would improve economic performance. In fact, they both argued that it should be bigger. On the other hand, James Buchanan, Edward Prescott, and Vernon Smithall Nobel laureates in economicsargued that the stimulus would not improve economic performance enough to justify the investment, saying that notwithstanding reports that all economists are now Keynesians... it is a triumph of hope over experience to believe that more government spending will help the US today. This was not an argument about precise quantities, but a disagreement about the policys basic effects.

Although fierce debates can be found in frontier areas of all sciences, this one would be as if, on the night before the Apollo moon launch, numerous Nobel laureates in physics were asserting that rockets couldnt get as far as the moon, almost as many were saying they could get there in theory but we need much more fuel, and some were arguing that the moon did not exist. The only thing an observer could say with high confidence before the stimulus program launched was that at least several Nobel laureates in economics would be directionally incorrect about its effects.

But the stimulus situation was even worse. It was clear at the time that we would not know which of them were right or wrong even after the fact. Suppose Professor Famous Economist X predicted on February 1, 2009, that unemployment will be about 10 percent in two years without the bill, and about 8 percent with the bill. What do you think would happen when 2011 rolled around and unemployment was 10 percent? Its a very, very safe bet that Professor X would say something like, Yes, but other conditions deteriorated faster than anticipated, so if we hadnt passed the stimulus bill, unemployment would have been more like 12 percent. So you see, I was right after all; it reduced unemployment by about 2 percentage points.