Bengfort Benjamin - Data analytics with Hadoop: an introduction for data scientists

by Benjamin Bengfort and Jenny Kim

Copyright 2016 Jenny Kim and Benjamin Bengfort. All rights reserved.

Printed in the United States of America.

Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

• Editor: Nicole Tache
• Production Editor: Melanie Yarbrough
• Copyeditor: Colleen Toporek
• Proofreader: Jasmine Kwityn
• Indexer: WordCo Indexing Services
• Interior Designer: David Futato
• Cover Designer: Randy Comer
• Illustrator: Rebecca Demarest

• June 2016: First Edition

See http://oreilly.com/catalog/errata.csp?isbn=9781491913703 for release details.

The OReilly logo is a registered trademark of OReilly Media, Inc. Data Analytics with Hadoop, the cover image, and related trade dress are trademarks of OReilly Media, Inc.

While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-491-91370-3

[LSI]

The term big data has come into vogue for an exciting new set of tools and techniques for modern, data-powered applications that are changing the way the world is computing in novel ways. Much to the statisticians chagrin, this ubiquitous term seems to be liberally applied to include the application of well-known statistical techniques on large datasets for predictive purposes. Although big data is now officially a buzzword, the fact is that modern, distributed computation techniques are enabling analyses of datasets far larger than those typically examined in the past, with stunning results.

Distributed computing alone, however, does not directly lead to data science. Through the combination of rapidly increasing datasets generated from the Internet and the observation that these data sets are able to power predictive models (more data is better than better algorithms), data products have become a new economic paradigm. Stunning successes of data modeling across large heterogeneous datasetsfor example, Nate Silvers seemingly magical ability to predict the 2008 election using big data techniqueshas led to a general acknowledgment of the value of data science, and has brought a wide variety of practitioners to the field.

Hadoop has evolved from a cluster-computing abstraction to an operating system for big data by providing a framework for distributed data storage and parallel computation. Spark has built upon those ideas and made cluster computing more accessible to data scientists. However, data scientists and analysts new to distributed computing may feel that these tools are programmer oriented rather than analytically oriented. This is because a fundamental shift needs to occur in thinking about how we manage and compute upon data in a parallel fashion instead of a sequential one.

This book is intended to prepare data scientists for that shift in thinking by providing an overview of cluster computing and analytics in a readable, straightforward fashion. We will introduce most of the concepts, tools, and techniques involved with distributed computing for data analysis and provide a path for deeper dives into specific topics areas.