Anthony Benoy - Professional Hadoop

Professional Hadoop

Published by

John Wiley & Sons, Inc.

10475 Crosspoint Boulevard

Indianapolis, IN 46256

www.wiley.com

Copyright 2016 by John Wiley & Sons, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 978-1-119-26717-1

ISBN: 978-1-119-26718-8 (ebk)

ISBN: 978-1-119-26720-1 (ebk)

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifi cally disclaim all warranties, including without limitation warranties of fi tness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services.

If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or Web site may provide or recommendations it may make. Further, readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Control Number: 2016934264

Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Programmer to Programmer, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affi liates, in the United States and other countries, and may not be used without written permission. Hadoop and Apache Hadoop are registered trademarks of The Apache Software Foundation. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc., is not associated with any product or vendor mentioned in this book.

BENOY ANTONY is an Apache Hadoop committer and has contributed features related to security and HDFS. He is the founder of DataApps (http://dataApps.io), a company that specializes in creating applications for big data. He maintains a Hadoop Security wiki at http://HadoopSecurity.org. Benoy is a Hadoop architect at eBay where he focuses on enhancing security and availability on eBay's Hadoop clusters without limiting user productivity. He regularly speaks at conferences like Hadoop Summit.

DR. KONSTANTIN BOUDNIK, co-founder and CEO of Memcore.io, is one of the early developers of Hadoop and a co-author of Apache Bigtop, the open source framework and the community around creation of software stacks for data processing projects. With more than 20 years of experience in software development, big- and fast-data analytic, Git, distributed systems and more, Dr. Boudnik has authored 15 US patents in distributed computing. Dr. Boudnik contributed to a dozen of open source projects in the area of distributed computing and data processing. He has helped and championed a number of successful Apache projects in the area.

CHERYL ADAMS is a senior cloud data and infrastructure architect. Her work includes supporting healthcare data for large government contracts; deploying production-based changes through scripting, monitoring, and troubleshooting; and monitoring environments using the latest tools for databases, web servers, web API, and storage.

BRANKY SHAO is a software engineer at eBay where he is building real time applications with Elasticsearch, Cassandra, Kafka, and Storm. He has been working with the Hadoop ecosystem technologies since 2010. He has extensive experience designing and implementing various software including distributed systems, data integration, framework/APIs, and web applications. He is passionate about open source and is a contributor to the Cascading project.

CAZEN LEE is a software architect at Samsung SDS. He is currently in charge of the Hadoop module for Samsung's big data platform. Prior to joining Samsung, Cazen served as a developer and architect for the integrated data warehouse layer in the financial industry, including work with Samsung Life Insurance and Korea Securities Finance Corp. He is also interested in both machine learning and neural network models.

KAI SASAKI is a Japanese software engineer who is interested in distributed computing and machine learning. Currently he is working at Treasure Data Inc., launched by Japanese entrepreneurs based in Silicon Valley. Although the beginning of his career didn't start with Hadoop or Spark, his interest in middleware and the fundamental technologies that support a lot of these types of big data services and the Internet drove him toward this field. He has been a Spark contributor, developing mainly MLlib and ML libraries. Nowadays, he is trying to research the great potential of combining deep learning and big data. He believes that Spark can play a significant role even in artificial intelligence within the big data era. You can find him on GitHub at https://github.com/Lewuathe.

Snehal Nagmote is a staff software engineer for the search infrastructure team at Walmart Labs. Some of his responsibilities include building data platform applications using the big data stack, and using tools such as Hadoop, Hive, Kafka, Flume, and Spark. Currently, he is focusing on building a near real time indexing data pipeline using Spark Streaming and Kafka.