Eliezer Yudkowsky - Inadequate Equilibria: Where and How Civilizations Get Stuck

Inadequate Equilibria is a book about a generalized notion of efficient markets, and how we can use this notion to guess where society will or wont be effective at pursuing some widely desired goal.

An efficient market is one where smart individuals should generally doubt that they can spot overpriced or underpriced assets. We can ask an analogous question, however, about the efficiency of other human endeavors.

Suppose, for example, that someone thinks they can easily build a much better and more profitable social network than Facebook, or easily come up with a new treatment for a widespread medical condition. Should they question whatever clever reasoning led them to that conclusion, in the same way that most smart individuals should question any clever reasoning that causes them to think AAPL stock is underpriced? Should they question whether they can beat the market in these areas, or whether they can even spot major in-principle improvements to the status quo? How efficient, or adequate, should we expect civilization to be at various tasks?

There will be, as always, good ways and bad ways to reason about these questions; this book is about both. I think that generalized notions of efficiency are the primary key to good reasoning about these real-life questions. And it is often wiser to spend more time thinking about good reasoning than bad reasoning, and to lay the good foundations first. So generalized notions of efficiency and inefficiency take priority in explanation; they are the topic of the first half of this book.

The second half of this book will then delve into background questions of mindset and methodology going into inadequacy analysis, and ways that this analysis can go wrong, particularly for the underconfident.

Several of my co-workers have been invaluable to the task of editing this book: Rob Bensinger, Matthew Graves, Jimmy Rintjema, and Nate Soares. I am also grateful to Scott Aaronson, Michael Arc, Stuart Armstrong, Bryan Caplan, Andrew Critch, Spencer Greenberg, Robin Hanson, Roxanne Heston, Ben Hoffman, Holden Karnofsky, Michael Keenan, Dan Keys, Peter McCluskey, Alex Mennen, Luke Muehlhauser, Toby Ord, Anna Salamon, Buck Shlegeris, Carl Shulman, Alyssa Vance, Qiaochu Yuan, and the many other thoughtful reviewers who provided comments and critiques. Any remaining inadequacies in this volume are my own.

This is a book about two incompatible views on the age-old question: When should I think that I may be able to do something unusually well?

These two viewpoints tend to give wildly different, nearly cognitively nonoverlapping analyses of questions like:

• My doctor says I need to eat less and exercise, but a lot of educated-sounding economics bloggers are talking about this thing called the Shangri-La Diet. Theyre saying that in order to lose weight, all you need to do is consume large quantities of flavorless, high-calorie foods at particular times of day; and they claim some amazing results with this diet. Could they really know better than my doctor? Would I be able to tell if they did?

• My day job is in artificial intelligence and decision theory. And I recall the dark days before 2015, when there was plenty of effort and attention going into advancing the state of the art in AI capabilities, but almost none going into AI alignment: better understanding AI designs and goals that can safely scale with capabilities. Though interest in the alignment problem has since increased quite a bit, it still makes sense to ask whether at the time I should have inferred from the lack of academic activity that there was no productive work to be done here; since if there were reachable fruits, wouldnt academics be taking them?

• Should I try my hand at becoming an entrepreneur? Whether or not it should be difficult to spot promising ideas in a scientific field, it certainly cant be easy to think up a profitable idea for a new startup. Will I be able to find any good ideas that arent already taken?

• The effective altruism community is a network of philanthropists and researchers that try to find the very best ways to benefit others per dollar, in full generality. Where should effective altruism organizations like GiveWell expect to find low-hanging fruitneglected interventions ripe with potential? Where should they look to find things that our civilization isnt already doing about as well as can be done?

When I think about problems like these, I use what feels to me like a natural generalization of the economic idea of efficient markets. The goal is to predict what kinds of efficiency we should expect to exist in realms beyond the marketplace, and what we can deduce from simple observations. For lack of a better term, I will call this kind of thinking inadequacy analysis.

Toward the end of this book, Ill try to refute an alternative viewpoint that is increasingly popular among some of my friends, one that I think is ill-founded. This viewpoint is the one Ive previously termed modesty, and the message of modesty tends to be: You cant expect to be able to do X that isnt usually done, since you could just be deluding yourself into thinking youre better than other people.

Ill open with a cherry-picked example that I think helps highlight the difference between these two viewpoints.