John Kelleher - Deep Learning

The MIT Press Essential Knowledge Series

A complete list of the titles in this series appears at the back of this book.

John D. Kelleher

The MIT Press | Cambridge, Massachusetts | London, England

2019 The Massachusetts Institute of Technology

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

This book was set in Chaparral Pro by Toppan Best-set Premedia Limited. Printed and bound in the United States of America.

Library of Congress Cataloging-in-Publication Data

Names: Kelleher, John D., 1974- author.

Title: Deep learning / John D. Kelleher.

Description: Cambridge, MA : The MIT Press, [2019] | Series: The MIT press essential knowledge series | Includes bibliographical references and index.

Identifiers: LCCN 2018059550 | ISBN 9780262537551 (pbk. : alk. paper)

Subjects: LCSH: Machine learning. | Artificial intelligence.

Classification: LCC Q325.5 .K454 2019 | DDC 006.3/1dc23 LC record available at https://lccn.loc.gov/2018059550

10 9 8 7 6 5 4 3 2 1

d_r0

The MIT Press Essential Knowledge series offers accessible, concise, beautifully produced pocket-size books on topics of current interest. Written by leading thinkers, the books in this series deliver expert overviews of subjects that range from the cultural and the historical to the scientific and the technical.

In todays era of instant information gratification, we have ready access to opinions, rationalizations, and superficial descriptions. Much harder to come by is the foundational knowledge that informs a principled understanding of the world. Essential Knowledge books fill that need. Synthesizing specialized subject matter for nonspecialists and engaging critical topics through fundamentals, each of these compact volumes offers readers a point of access to complex ideas.

Bruce Tidor

Professor of Biological Engineering and Computer Science

Massachusetts Institute of Technology

Deep learning is enabling innovation and change across all aspects of our modern lives. Most of the artificial intelligence breakthroughs that you hear about in the media are based on deep learning. As a result, whether you are a business person interested in improving the efficiency of your organization, a policymaker concerned with ethics and privacy in a Big Data world, a researcher working with complex data, or a curious citizen who wants a better sense of the potential of artificial intelligence and how it will change your life, it is important for you to have an understanding of deep learning.

The goal of this book is to enable the general reader to gain an understanding of what deep learning is, where it has come from, how it works, what it makes possible (and what it doesnt), and how the field is likely to develop in the next ten years. The fact that deep learning is a set of algorithms and models means that understanding deep learning requires understanding how these algorithms and models process data. As a result, this book is not purely descriptive and definitional; it also includes explanations of algorithms. I have attempted to present the technical material in an accessible way. From my teaching experience, I have found that for technical topics the most accessible presentation is to explain the fundamental concepts in a step-by-step manner. So, although I have tried to keep the mathematical content to a minimum, where I felt it was necessary to include it I have endeavored to walk you through the mathematical equations in as clear and direct a manner as I can. I have supplemented these explanations with examples and illustrations.

What is really wondrous about deep learning is not the complexity of the math it is built on, but rather, that it can perform such a diverse set of exciting and impressive tasks using such simple calculations. Dont be surprised to find yourself saying: Is that all its doing? In fact, a deep learning model really is just a lot (admittedly, an awful lot) of multiplications and additions with a few nonlinear mappings (which I will explain) added in. Yet, despite this simplicity, these models can, among other achievements, beat the Go world champion, define the state-of-the-art in computer vision and machine translation, and drive a car. This book is an introductory text about deep learning, but I hope that it is an introduction that has enough depth that you will come back to the book as your confidence with the material grows.

This book would not have been possible without the sacrifices made by my wife, Aphra, and my family, in particular my parents John and Betty Kelleher. I have also received a huge amount of support from friends, especially Alan McDonnell, Ionela Lungu, Simon Dobnik, Lorraine Byrne, Noel Fitzpatrick, and Josef van Genabith.

I would also like to acknowledge the help I have received from the staff at the MIT Press, and from a number of people who have read sections of the book and provided feedback. MIT Press organized three anonymous reviewers who read and commented on a draft of the book. I thank these reviewers for their time and helpful feedback. Also a number of people read draft chapters from the book and I wish to take this opportunity to publicly acknowledge their help, so my thanks to: Mike Dillinger, Magdalena Kacmajor, Elizabeth Kelleher, John Bernard Kelleher, Aphra Kerr, Filip Klubika, and Abhijit Mahalunkar. This book has been informed by the many conversations I have had with colleagues and students about deep learning, in particular those with Robert Ross and Giancarlo Salton.

This book is dedicated to my sister Elizabeth (Liz) Kelleher in recognition of her love and support, and her patience with a brother who cant stop explaining things.

Deep learning is the subfield of artificial intelligence that focuses on creating large neural network models that are capable of making accurate data-driven decisions. Deep learning is particularly suited to contexts where the data is complex and where there are large datasets available. Today most online companies and high-end consumer technologies use deep learning. Among other things, Facebook uses deep learning to analyze text in online conversations. Google, Baidu, and Microsoft all use deep learning for image search, and also for machine translation. All modern smart phones have deep learning systems running on them; for example, deep learning is now the standard technology for speech recognition, and also for face detection on digital cameras. In the healthcare sector, deep learning is used to process medical images (X-rays, CT, and MRI scans) and diagnose health conditions. Deep learning is also at the core of self-driving cars, where it is used for localization and mapping, motion planning and steering, and environment perception, as well as tracking driver state.

Perhaps the best-known example of deep learning is DeepMinds AlphaGo. Go is a board game similar to Chess. AlphaGo was the first computer program to beat a professional Go player. In March 2016, it beat the top Korean professional, Lee Sedol, in a match watched by more than two hundred million people. The following year, in 2017, AlphaGo beat the worlds No. 1 ranking player, Chinas Ke Jie.

In 2016 AlphaGos success was very surprising. At the time, most people expected that it would take many more years of research before a computer would be able to compete with top level human Go players. It had been known for a long time that programming a computer to play Go was much more difficult than programming it to play Chess. There are many more board configurations possible in Go than there are in Chess. This is because Go has a larger board and simpler rules than Chess. There are, in fact, more possible board configurations in Go than there are atoms in the universe. This massive search space and Gos large branching factor (the number of board configurations that can be reached in one move) makes Go an incredibly challenging game for both humans and computers.