Stefan H. Thomke - Experimentation Works: The Surprising Power of Business Experiments

Being an effective leader today means acting with a sense of urgency: making many decisions quickly under great uncertainty. Experimentation Works describes a vital methodologylarge-scale business experimentationthat takes the madness out of that equation and sets you and your organization up for success.

AJAY BANGA, President and CEO, Mastercard

Experimentation Works is required reading for all leaders who want to know how companies like Amazon, Google, and Netflix win through fast-cycle scientific experimentsand how it can be done in their organizations. Making business decisions without experimentation is like bungee jumping without testing the cord. Yet far too many organizations still do this. With Thomkes clear explanations and examples, yours wont be one of them.

SCOTT COOK, cofounder and Chairman of the Executive Committee, Intuit

Stefan Thomke has written a marvelous book thats both thought provoking and highly informative. As CEO, I saw how his research on business experimentation can transform innovation cultures for the better. Experimentation does indeed work. I highly recommend the book.

MARIJN DEKKERS, Chairman, Unilever

Experimentation Works is a masterpiece and a must-read. Thomke, a top academic authority with deep practical insights, explains how low-cost business experiments can revolutionize the way firms design everything from business models to customer experiences. The books ideas are very powerful and rigorous and will change how you think about the science and practice of innovation management.

ERIC VON HIPPEL, Professor, Massachusetts Institute of Technology; author, Democratizing Innovation

Stefan Thomke has written the definitive handbook for executives on how to build a high-impact experimentation capability. Experimentation Works is an amazing book with some sage advice: Experiment or perish. Its absolutely the right message at the right time in a world that has gone digital.

JAY LARSON, CEO, Optimizely

Business breakthroughs are achieved through the combination of great ideas and effective implementation. Stefan Thomkes work, which I applied enthusiastically during my tenure as CEO, demonstrates that a process of disciplined experimentation is the best way to drive innovation and gain sustained competitive advantage.

GARY LOVEMAN, former Chairman and CEO, Caesars Entertainment Corporation

Experimentation Works presents a very compelling case for how companies can leverage the scientific method to drive innovation and reinvent business models for sustainable growth. An immensely valuable guide for leaders of both B2C and B2B companies in an era of accelerating change and disruption.

LOH CHIN-HUA, CEO, Keppel Corporation

Stefan Thomke has written a rare book, one that combines deep and original insights with great practical advice. Experimentation Works is about revolutionary changes in how innovation is managed, decisions are made, and organizations are led. Thomke shows us how business experiments fuel innovation and why companies must test at large scale to compete. The book is a landmark achievement.

ANAND MAHINDRA, Chairman, Mahindra Group

In an increasingly digital world, CEOs need to drive large-scale scientific business experimentation. If they dont, their companies will die. Companies with strong experimentation systems and cultures will win, every time. If you want to learn how this is done, you must read Experimentation Works. The book will change how you think about managing and decision making.

MARK OKERSTROM, President and CEO, Expedia Group

EXPERIMENTATION WORKS

EXPERIMENTATION WORKS

THE SURPRISING POWER of BUSINESS EXPERIMENTS

STEFAN H. THOMKE

HARVARD BUSINESS REVIEW PRESS

BOSTON, MASSACHUSETTS

HBR Press Quantity Sales Discounts

Harvard Business Review Press titles are available at significant quantity discounts when purchased in bulk for client gifts, sales promotions, and premiums. Special editions, including books with corporate logos, customized covers, and letters from the company or CEO printed in the front matter, as well as excerpts of existing books, can also be created in large quantities for special needs.

For details and discount information for both print and ebook formats, contact .

Copyright 2020 Stefan H. Thomke

Cartoons by Tom Fishburne, Marketoonist, LLC.

All rights reserved

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

No part of this publication may be reproduced, stored in or introduced into a retrieval system, or transmitted, in any form, or by any means (electronic, mechanical, photocopying, recording, or otherwise), without the prior permission of the publisher. Requests for permission should be directed to , or mailed to Permissions, Harvard Business School Publishing, 60 Harvard Way, Boston, Massachusetts 02163.

The web addresses referenced in this book were live and correct at the time of the books publication but may be subject to change.

Library of Congress Cataloging-in-Publication Data

Names: Thomke, Stefan H., author.

Title: Experimentation works : the surprising power of business experiments / Stefan Thomke.

Description: Boston, Massachusetts : Harvard Business Review Press, [2019] | Includes index.

Identifiers: LCCN 2019030882 | ISBN 9781633697102 (hardcover) | ISBN 9781633697119 (ebook)

Subjects: LCSH: BusinessTechnological innovations. | Marketing research. | Creative ability in business. | ConsumersResearch. | Success in business.

Classification: LCC HD45 .T46 2019 | DDC 658.4/063dc23

LC record available at https://lccn.loc.gov/2019030882

ISBN: 978-1-63369-710-2

eISBN: 978-1-63369-711-9

The paper used in this publication meets the requirements of the American National Standard for Permanence of Paper for Publications and Documents in Libraries and Archives Z39.48-1992.

To my parents,
who got me started,
and Savita, Arjun, Vikram, and Anjali,
who keep me going

CONTENTS

Why I wrote the book (and why you should read it). Understanding the power of thinking and acting scientifically. Entering a new age of large-scale business experimentation.

The need for business experiments to drive innovation and profitable growth. Deploying large-scale experimentation as a businesswide practice. Brief introduction to the books ideas and frameworks.

The role of business experimentation in innovation. Understanding the power of tools and the experimentation process. Leveraging the operational drivers of high-velocity learning.

The elements of a good business experiment. Seven questions that yield better management decisions. Appreciating the limits of experimentation.

The business value of A/B testing. Leveraging the power of incremental innovation for business performance. Learning the best experimentation practices from leading digital companies.

The seven attributes of a true experimentation culture. Diagnosing and addressing cultural obstacles. Adopting a new management model for experimentation organizations.

The operating model of a true experimentation organization. Democratizing testing through process, management, and cultural discipline. Using technology, scale, and velocity for competitive advantage.

The steps to becoming a true experimentation organization. Using seven system levers and the ABCDE maturity framework to analyze your situation. Deploying experimentation tools most effectively.

The myths that undermine experimentation and innovation. Realizing that your actions will lead to opposite reactions. Addressing fallacies that slow down progress.