Michael Schrage - Recommendation Engines

The MIT Press Essential Knowledge Series

A complete list of the titles in this series appears at the back of this book.

Michael Schrage

The MIT Press | Cambridge, Massachusetts | London, England

2020 Massachusetts Institute of Technology

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

This book was set in Chaparral Pro by New Best-set Typesetters Ltd.

Library of Congress Cataloging-in-Publication Data

Names: Schrage, Michael, author.

Title: Recommendation engines / Michael Schrage.

Description: Cambridge, Massachusetts : The MIT Press, 2020. | Series: The MIT Press essential knowledge series | Includes bibliographical references and index.

Identifiers: LCCN 2019042167 | ISBN 9780262539074 (paperback)

Subjects: LCSH: Recommender systems (Information filtering)

Classification: LCC ZA3084 .S37 2020 | DDC 025.04dc23

LC record available at https://lccn.loc.gov/2019042167

10 9 8 7 6 5 4 3 2 1

d_r0

The MIT Press Essential Knowledge series offers accessible, concise, beautifully produced pocket-size books on topics of current interest. Written by leading thinkers, the books in this series deliver expert overviews of subjects that range from the cultural and the historical to the scientific and the technical.

In todays era of instant information gratification, we have ready access to opinions, rationalizations, and superficial descriptions. Much harder to come by is the foundational knowledge that informs a principled understanding of the world. Essential Knowledge books fill that need. Synthesizing specialized subject matter for nonspecialists and engaging critical topics through fundamentals, each of these compact volumes offers readers a point of access to complex ideas.

Who doesnt likeor needgreat advice? Who doesnt appreciate the timely and thoughtful suggestion that pleasantly surprises? When I recall pivotal moments in my lifepersonal and professionalI cant help but remember the impact and influence recommendation and advice had on my decisions. Some suggestions were breathtaking in their awfulnessas if my friend or colleague were totally clueless about me; others were so sharp they startled me to self-awareness and insight I would never have come to on my own. Why didnt I think of that?

Of course the good in good advice is firmly in the mind of the recipient. Ive long wondered how my own advice and recommendations were taken, particularly professionally: was I truly being helpful? Or was my counsel more self-indulgent than constructive? How could I have made my suggestion better or more appealing? Pun intended, Ive always takenand givenadvice seriously. I know how importanthow transformativethe right recommendation at the right time can be.

The right recommendation at the right time is exactly what an Amazon, a Netflix, a Facebook, a Spotify, a Google, a LinkedIn, a Tinder, a TikTok, and a YouTube aspire to. That makes their ambitions transformative. Their massive, high-powered recommendation engines, also called recommendation engines, not only algorithmically anticipate what people like you desire, they nudge users to explore options and opportunities that might never have crossed their minds.

We learned what worked and what didnt by seeing how changes in the recommendations helped people find new books, recalls Greg Linden, who pioneered several of Amazons earliest recommenders and was a great help for this book. We enjoyed helping people discover books they probably would not have found on their own. It was never about marketingjust matching people to books they would love. But it turns out people do buy more when you help them find what they need.

Indeed. But what happens in a world where that digital device in your hand or the Airpod in your ear instantly and reliably gives you better advice than your best friend? This book describes what recommendations make that world go round.

This is a dispassionate passion project: the purpose and power of recommendation fascinates me intellectually and emotionally. It demands exploration. The way a sharp piece of advice or (seemingly) casual recommendation can change someones life is awesome and compelling. Reason and feeling converge. Ideally, life and its options get better.

Yet peoples relationship with recommendation and advice is as poorly understood as it is underappreciated. The sciencecognitive psychology, neurophysiology, behavioral economicsis not settled. As digital media and algorithmic innovation increasingly define and determine how people get and experience advice, we must review and revise what recommendation means.

Bluntly, technology is taking over advice. For some critics, this takeover is hostile; for others, it offers the data-driven gift of intimacy, immediacy, and discovery. Sooner rather than later, most people in most places wont make decisions about what to do or where to go or who to contact without revving up a recommendation engine. They want their recommenders giving them greater confidence and clarity around what they want whenever they want. Thats powerful. It charts a future trajectory of human behavior.

More by happenstance than by design, Ive been directly and indirectly involved with the rise of recommendation engines since their academic origins at MITnotably at the Media Lab and Sloan School. While sensing their importance in the moment (my colleagues are smart people), I confess to not thinking big enough. I didnt anticipate clever prototypes hacked with simple statistical rules and semistructured databases scaling so quickly into multidisciplinary juggernauts of big data and machine learning innovation. My imagination failed me. Id no idea recommendation would evolve from a nifty little start-up feature in software to an organizing principle for commercialand globalsystems design. Stupid me. This book is penance.

Remarkable, isnt it, how trillion-dollar market potential and the power to influence both pop culture and purchasing worldwide can motivate entrepreneurial research? Jeff Bezos, Jack Ma, Reid Hoffman, Katrina Lake, Daniel Ek, Mark Zuckerberg, and Reid Hastings didnt succeed at digital disruption by accident; their organizations delivered recommendations that fundamentally transformed customer expectations and experiences around choice.

This books purpose is exploring and explaining the essential knowledge of this ongoing transformation. History is as important as technology here; humans are as vital as the machines. In fact, recommenders gleefully mash up the most interesting aspects and elements of the mind with the most challenging and controversial components of networked artificial intelligence. Theyre both personalization and prediction engines. What, exactly, makes a tune catchy for you? Why do you follow this suggestion but not that one? Recommenders breed inferential interdependence. One cant appreciate the recommendation future without grasping how machinesand their humansturn data into decisions. Theres no escaping either the technology or its underlying mathematics.

Once upon a time, for example, people used maps, a compass, guidebooks, and the stars to navigate the world; today, multimedia medleys of GPS, animated high-resolution displays, and sultry female voices literally suggest people turn left if they want to save seventeen minutes of transit time or veer right should they want incredible views (and, by the way, the restaurant in 3.2 kilometers has the worlds best pancakes). Recommendation, advice, and their associated optionality inextricably and increasingly entwine in ways people dont just expect but have come to demand.