Stuart Russell - Human Compatible: Artificial Intelligence and the Problem of Control

A long time ago, my parents lived in Birmingham, England, in a house near the university. They decided to move out of the city and sold the house to David Lodge, a professor of English literature. Lodge was by that time already a well-known novelist. I never met him, but I decided to read some of his books: Changing Places and Small World. Among the principal characters were fictional academics moving from a fictional version of Birmingham to a fictional version of Berkeley, California. As I was an actual academic from the actual Birmingham who had just moved to the actual Berkeley, it seemed that someone in the Department of Coincidences was telling me to pay attention.

One particular scene from Small World struck me: The protagonist, an aspiring literary theorist, attends a major international conference and asks a panel of leading figures, What follows if everyone agrees with you? The question causes consternation, because the panelists had been more concerned with intellectual combat than ascertaining truth or attaining understanding. It occurred to me then that an analogous question could be asked of the leading figures in AI: What if you succeed? The fields goal had always been to create human-level or superhuman AI, but there was little or no consideration of what would happen if we did.

A few years later, Peter Norvig and I began work on a new AI textbook, whose first edition appeared in 1995. The books final section is titled What If We Do Succeed? The section points to the possibility of good and bad outcomes but reaches no firm conclusions. By the time of the third edition in 2010, many people had finally begun to consider the possibility that superhuman AI might not be a good thingbut these people were mostly outsiders rather than mainstream AI researchers. By 2013, I became convinced that the issue not only belonged in the mainstream but was possibly the most important question facing humanity.

In November 2013, I gave a talk at the Dulwich Picture Gallery, a venerable art museum in south London. The audience consisted mostly of retired peoplenonscientists with a general interest in intellectual mattersso I had to give a completely nontechnical talk. It seemed an appropriate venue to try out my ideas in public for the first time. After explaining what AI was about, I nominated five candidates for biggest event in the future of humanity:

1. We all die (asteroid impact, climate catastrophe, pandemic, etc.).

2. We all live forever (medical solution to aging).

3. We invent faster-than-light travel and conquer the universe.

4. We are visited by a superior alien civilization.

5. We invent superintelligent AI.

I suggested that the fifth candidate, superintelligent AI, would be the winner, because it would help us avoid physical catastrophes and achieve eternal life and faster-than-light travel, if those were indeed possible. It would represent a huge leapa discontinuityin our civilization. The arrival of superintelligent AI is in many ways analogous to the arrival of a superior alien civilization but much more likely to occur. Perhaps most important, AI, unlike aliens, is something over which we have some say.

Then I asked the audience to imagine what would happen if we received notice from a superior alien civilization that they would arrive on Earth in thirty to fifty years. The word pandemonium doesnt begin to describe it. Yet our response to the anticipated arrival of superintelligent AI has been... well, underwhelming begins to describe it. (In a later talk, I illustrated this in the form of the email exchange shown in .) Finally, I explained the significance of superintelligent AI as follows: Success would be the biggest event in human history... and perhaps the last event in human history.

From: Superior Alien Civilization

To: humanity@UN.org

Subject: Contact

Be warned: we shall arrive in 3050 years

From: humanity@UN.org

To: Superior Alien Civilization

Subject: Out of office: Re: Contact

Humanity is currently out of the office. We will respond to your message when we return.

FIGURE 1 : Probably not the email exchange that would follow the first contact by a superior alien civilization.

A few months later, in April 2014, I was at a conference in Iceland and got a call from National Public Radio asking if they could interview me about the movie Transcendence, which had just been released in the United States. Although I had read the plot summaries and reviews, I hadnt seen it because I was living in Paris at the time, and it would not be released there until June. It so happened, however, that I had just added a detour to Boston on the way home from Iceland, so that I could participate in a Defense Department meeting. So, after arriving at Bostons Logan Airport, I took a taxi to the nearest theater showing the movie. I sat in the second row and watched as a Berkeley AI professor, played by Johnny Depp, was gunned down by anti-AI activists worried about, yes, superintelligent AI. Involuntarily, I shrank down in my seat. (Another call from the Department of Coincidences?) Before Johnny Depps character dies, his mind is uploaded to a quantum supercomputer and quickly outruns human capabilities, threatening to take over the world.