Michael Luca - The Power of Experiments: Decision Making in a Data-Driven World

• Advance praise for The Power of Experiments
• A tremendous achievement, and inspiring to boot. Leaders in government and industry take note: Luca and Bazermans book will guide you to better decisions in this age of experiments.
• Cass R. Sunstein, Robert Walmsley University Professor, Harvard Law School; coauthor of Nudge
• Luca and Bazerman tell the fascinating story of the rise of experiments in the tech sector and beyond, and deliver a powerful message: experiments are transforming how businesses and governments make decisions. Going only by gut isnt enoughsuccessful leaders need to move between data and decisions.
• Charles Duhigg, author of The Power of Habit and Smarter Faster Better
• The authors have created a masterpiece, a manifesto showcasing the deepest revolution in the social sciences in the past twenty-five yearsfield experimentation. This book is a must-read for anyone interested in understanding policy, behavioral economics, technology, and life itself.
• John A. List, Kenneth C. Griffin Distinguished Service Professor, University of Chicago
• When testing and experimentation are at the heart of a culture, a team or company can be more responsive and nimble, driving a better product and more growth alongside a better employee experience.
• Gillian Tans, Chairwoman, Booking.com
• One of the great things about e-commerce is that it is far easier to run experiments online than offline. As more and more companies move online, they need to learn how to use this powerful tool. This book shows how to take advantage of experiments and how this will revolutionize business, both online and off.
• Hal Varian, Chief Economist, Google
• This accessible and engaging book provides an excellent introduction to a subject that every young person entering the business world today should understandexperimentation. The case studies draw the reader into the challenges that arise in practice, highlighting issues ranging from bias to ethics to unintended consequences.
• Susan Athey, The Economics of Technology Professor, Stanford Graduate School of Business
• Leaders who create an environment for taking calculated risks get better results. This book reminds us that systemic experimentation and calculated risk taking have built great companies and great societies alike.
• Deval Patrick, former Governor of Massachusetts
• Luca and Bazermans The Power of Experiments will open your eyes about how to distill information from data.
• Alvin E. Roth, Craig and Susan McCaw Professor of Economics, Stanford University; recipient of the 2012 Nobel Memorial Prize in Economics
• So often we dont know what we dont know until we tryand the best form of trying is often an experiment. This book is a brilliant tour through the past, present, and future of experimentation, packed full of insight and examples. Read it and learnI did.
• Tim Harford, author of The Undercover Economist and presenter of Cautionary Tales

Michael Luca and Max H. Bazerman

The MIT Press

Cambridge, Massachusetts | London, England

2020 Michael Luca and Max H. Bazerman

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

This book was set in ITC Stone Serif Std and ITC Stone Sans Std by Toppan Best-set Premedia Limited.

Library of Congress Cataloging-in-Publication Data

Names: Luca, Michael, author. | Bazerman, Max H., author.

Title: The power of experiments : decision making in a data-driven world /

Michael Luca, Max H. Bazerman.

Description: Cambridge, Massachusetts : The MIT Press, [2020] | Includes

bibliographical references and index.

Identifiers: LCCN 2019029932 | ISBN 9780262043878 (hardcover) |

ISBN 9780262358255 (ebook)

Subjects: LCSH: Economics--Psychological aspects. | Experimental economics. |

Decision making. | Psychology.

Classification: LCC HB74.P8 L83 2020 | DDC 330.01/9--dc23

LC record available at https://lccn.loc.gov/2019029932

10 9 8 7 6 5 4 3 2 1

How many experiments do you think youve participated in over the past year? Were talking about randomized controlled trialsexperiments designed to test the impact of different treatments by randomly assigning you and other participants (often called subjects) to various treatment conditions, like those you might have participated in if you took Psych 101 or if youve tried out an experimental drug.

So, whats your number? At first blush, you might think the answer is zero. But unless you live in a bunker with no Internet access, youve likely participated in many experiments over the past year. If you logged onto Facebook right now, theres a good chance youd be an unwitting subject in a variety of the companys ongoing experiments as you scroll through your News Feed and peruse the ads being shown to you. You are also likely to be a test subject if you search for an item on Google, watch a movie on Netflix, respond to email surveys, or call companies for customer support.

In a dramatic departure from its historic role as an esoteric tool for academic research, the randomized controlled trial has gone mainstream. Historically, experimental methods may have been alien to the managerial toolkit. These days, companies like Google wouldnt dare make a major change in their platforms without first looking at experiments to understand how it would influence user behavior. From startups to international conglomerates to government agencies, organizations have a new tool to develop frameworks and test ideas, and to understand the impact of the products and services they are providing.

We are in the early days of the age of experiments. Experiments help to complement intuition and guesswork with evidence-based decision making. Weve seen many experiments lead to large returns for organizationsas when a simple experiment allowed eBay to learn that it was wasting millions of dollars per year in an advertising campaign. The discoveries that experiments generate are largely a good thing: good for governments, which can use experimental results to better design and deliver services; good for companies, which can use experimental results to improve their human resources practices and offer better products; and, when carefully designed and monitored, good for employees, customers, and citizens as well. Well highlight success stories arising from experiments and the lessons we can draw from them.

At the same time, weve seen plenty of experiments that were poorly executed or interpreted, yielding misleading conclusions or wasted time and effort. Even in the best of worlds, experiments can be challenging to design and interpretand require managerial judgment. For example, when gym chain 24 Hour Fitness ran experiments aimed at identifying ways to increase attendance rates, they found that many of the interventions that increased gym-going in the short run were often considerably less impactful when the results were tracked over several months.

Designing and interpreting experiments can be challenging tasks, requiring a unique combination of skills. In the age of experiments, effective managers increasingly need frameworks for going from experimental results to decisions. This book will help to develop and draw out principles that will help managers make the most of experimental results.