Ahmed Sherif - Apache Spark Deep Learning Cookbook: Over 80 recipes that streamline deep learning in a distributed environment with Apache Spark

Copyright 2018 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Commissioning Editor: Amey Varangaonkar
Acquisition Editor: Tushar Gupta
Content Development Editor: Snehal Kolte
Technical Editor: Dharmendra Yadav
Copy Editors: Safis Editing
Project Coordinator: Manthan Patel
Proofreader: Safis Editing
Indexer: Aishwarya Gangawane
Graphics: Jisha Chirayil
Production Coordinator: Nilesh Mohite

First published: July 2018

Production reference: 1090718

Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.

ISBN 978-1-78847-422-1

www.packtpub.com

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

• Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

• Improve your learning with Skill Plans built especially for you

• Get a free eBook or video every month

• Mapt is fully searchable

• Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.

At www.PacktPub.com , you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

If you are reading that, it's safe to assume that you are well-aware of the tremendous impact of Artificial Intelligence (AI) and Machine Learning (ML), and the uncanny effectiveness of deep neural networks. Matei Zaharia and his team started Spark, not as a competitor to Hadoop, but rather as an effort towards the democratization of AI and ML. As Zaharia has famously said, The only focus in Spark is how you compute stuff, not where you store it. Dubbed as unified analytics engine for large-scale data processing, Spark is optimized for resilience, speed, ease of use, generality, and run-everywhere features, and this book does a phenomenal job explaining it to you, converting you to a spark enthusiast.

As a reader, if you are excited about getting started with Spark's application in deep learning, this book can help. The authors begin by helping to set up Spark for Deep Learning development by providing clear and concisely written recipes. The initial setup is naturally followed by creating a neural network, elaborating on the pain points of convolutional neural networks, and recurrent neural networks. AI find new use cases every day, mostly starting with verticals. In the practitioner's spirit, authors provided practical (yet simplified) use cases of predicting fire department calls with SparkML, real estate value prediction using XGBoost, predicting the stock market cost of Apple with LSTM, and creating a movie recommendation engine with Keras.

The AI and ML landscape is nothing if not heterogeneous; the dizzying diversity of toolset and libraries can be intimidating. The authors do an excellent job mixing it up with different libraries, and incorporating relevant yet diverse technologies as the reader moves forward in the book. As deep learning frameworks start to converge and move up in abstraction, the scale of the exploratory data analysis inevitably grows. That's why instead of creating a proverbial one-trick pony (or YOLO model, pun intended), the book covers pertinent and highly relevant technologies such as LSTMs in Generative Networks, natural language processing with TF-IDF, face recognition using deep convolutional networks, creating and visualizing word vectors using Word2Vec and image classification with TensorFlow on Spark. Aside from crisp and focused writing, this wide array of highly relevant ML and deep learning topics give the book its core strength.

I have had a unique opportunity to see Spark in action, both in academia and in the industry. As a Stanford visiting scholar, I have attended various sessions by Matei Zaharia, the co-creator of Spark and enjoyed his vision concerning Databricks, the operationalization of algorithms at scale, and future of big data. At the same time as a practitioner and chief architect of AI and ML, I witness first-hand how world's largest retailer labs standardized their operational ML model deployments with Spark. As cloud enables new data application architectures, I see Spark as a key enabler and accelerator for new and unpreceded ML implementations across a variety of industry verticals. The rationale is quite simple; C-suite looks at cutting-edge modern technology to go beyond typical developer's hello world test or a science project. It needs to solve a real-world business problem at scale, have a thriving eco-system, and ability to provide enterprise-grade SLAs with around the clock support. Spark fulfill all these criteria; and goes beyond the basics with unprecedented speed, resilience, and community adaption. Spark is regularly used in advance analytics; use cases such as complex outlier analysis for system telemetry that observes and perceives meaningful patterns prior to these becoming unanticipated problems, or a missed opportunity as outliers can be both. Consider Spark as a tool that bridges the gap between traditional BI, and modern ML enabled services for delivering meaningful business outcomes. As data becomes the central competitive advantage, it is crucial to see disruptions before they occur, all while keeping account of seasonality, cyclicity, and ever-increasing temporal correlations. Ease of operationalization is the key to industry adaption, and that's where Spark shines like no other; data to insight, and insight to action.