Recorded Books Inc. - Competing against luck: the story of innovation and customer choice

Were lost, but were making good time!

Yogi Berra

This is a book about progress.

Yes, its a book about innovationand how to get better at it. But at its core, this book is about the struggles we all face to make progress in our lives.

If youre like many entrepreneurs and managers, the word progress might not spring to mind when youre trying to innovate. Instead you obsess about creating the perfect product with just the right combination of features and benefits to appeal to customers. Or you try to continually fine-tune your existing products so theyre more profitable or differentiated from your competitors. You think you know just what your customers would like, but in reality, it can feel pretty hit or miss. Place enough bets andwith a bit of lucksomething will work out.

But that doesnt have to be the case, not when you truly understand what causes consumers to make the choices they do. Innovation can be far more predictableand far more profitablebut only if you think about it differently. Its about progress, not products. So if you are tired of throwing yourself and your organization into well-intended innovation efforts that routinely underwhelm; if you want to create products and services that you know, in advance, customers will not only be eager to buy, but willing to pay a premium price for; if you want to competeand winagainst those relying on luck to successfully innovate, then read on. This book is about helping you make progress, too.

For as long as I can remember, innovation has been a top priorityand a top frustrationfor companies around the world. In a recent McKinsey poll, 84 percent of global executives acknowledged that innovation is extremely important to their growth strategies, yet a staggering 94 percent were unsatisfied with their own innovation performance. Most people would agree that the vast majority of innovations fall far short of ambitions, a fact that has remained unchanged for decades.

On paper, this makes no sense. Companies have never had more sophisticated tools and techniques at their disposaland there are more resources than ever deployed in reaching innovation goals. In 2015, according to an article in strategy + business, one thousand publicly held companies spent $680 billion on research and development alone, a 5.1 percent increase over the previous year.

And businesses have never known more about their customers. The big data revolution has greatly increased the variety, volume, and velocity of data collection, along with the sophistication of the analytical tools applied to it. Hopes for this data trove are higher than ever. Correlation is enough, then-Wired editor in chief Chris Anderson famously declared in 2008. We can, he implied, solve innovation problems by the sheer brute force of the data deluge. Ever since Michael Lewis chronicled the Oakland As unlikely success in Moneyball (who knew on-base percentage was a better indicator of offensive success than batting averages?), organizations have been trying to find the Moneyball equivalent of customer data that will lead to innovation success. Yet few have.

Innovation processes in many companies are structured and disciplined, and the talent applying them is highly skilled. There are careful stage-gates, rapid iterations, and checks and balances built into most organizations innovation processes. Risks are carefully calculated and mitigated. Principles like six-sigma have pervaded innovation process design so we now have precise measurements and strict requirements for new products to meet at each stage of their development. From the outside, it looks like companies have mastered an awfully precise, scientific process.

But for most of them, innovation is still painfully hit or miss. And worst of all, all this activity gives the illusion of progress, without actually causing it. Companies are spending exponentially more to achieve only modest incremental innovations while completely missing the mark on the breakthrough innovations critical to long-term, sustainable growth. As Yogi Berra famously observed: Were lost, but were making good time!

Whats gone so wrong?

Here is the fundamental problem: the masses and masses of data that companies accumulate are not organized in a way that enables them to reliably predict which ideas will succeed. Instead the data is along the lines of this customer looks like that one, this product has similar performance attributes as that one, and these people behaved the same way in the past, or 68 percent of customers say they prefer version A over version B. None of that data, however, actually tells you why customers make the choices that they do.

Let me illustrate. Here I am, Clayton Christensen. Im sixty-four years old. Im six feet eight inches tall. My shoe size is sixteen. My wife and I have sent all our children off to college. I live in a suburb of Boston and drive a Honda minivan to work. I have a lot of other characteristics and attributes. But these characteristics have not yet caused me to go out and buy the New York Times today. There might be a correlation between some of these characteristics and the propensity of customers to purchase the Times. But those attributes dont cause me to buy that paperor any other product.

If a company doesnt understand why I might choose to hire its product in certain circumstancesand why I might choose something else in othersits data is unlikely to help it create any new innovations for me. Its seductive to believe that we can see important patterns and cross-references in our data sets, but that doesnt mean one thing actually caused the other. As Nate Silver, author of The Signal and the Noise: Why So Many Predictions FailBut Some Dont, points out, ice cream sales and forest fires are correlated because both occur more often in the summer heat. But there is no causation; you dont light a patch of the Montana brush on fire when you buy a pint of Hagen-Dazs.

Of course, its no surprise that correlation isnt the same as causality. But although most organizations know that, I dont think they act as if there is a difference. Theyre comfortable with correlation. It allows managers to sleep at night.

But correlation does not reveal the one thing that matters most in innovationthe causality behind why I might purchase a particular solution. Yet few innovators frame their primary challenge around the discovery of a cause. Instead, they focus on how they can make their products better, more profitable, or differentiated from the competition.

As W. Edwards Deming, the father of the quality movement that transformed manufacturing, once said: If you do not know how to ask the right question, you discover nothing. After decades of watching great companies fail over and over again, Ive come to the conclusion that there is, indeed, a better question to ask: What job did you hire that product to do?

For me, this is a neat idea. When we buy a product, we essentially hire something to get a job done. If it does the job well, when we are confronted with the same job, we hire that same product again. And if the product does a crummy job, we fire it and look around for something else we might hire to solve the problem.

Every day stuff happens to us. Jobs arise in our lives that we need to get done. Some jobs are little (pass the time while waiting in line), some are big (find a more fulfilling career). Some surface unpredictably (dress for an out-of-town business meeting after the airline lost my suitcase), some regularly (pack a healthy, tasty lunch for my daughter to take to school). Other times we know theyre coming. When we realize we have a job to do, we reach out and pull something into our lives to get the job done. I might, for example, choose to buy the