Hadoop Security
by Ben Spivey and Joey Echeverria
Copyright 2015 Joseph Echeverria and Benjamin Spivey. All rights reserved.
Printed in the United States of America.
Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.
OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .
- Editors: Ann Spencer and Marie Beaugureau
- Production Editor: Melanie Yarbrough
- Copyeditor: Gillian McGarvey
- Proofreader: Jasmine Kwityn
- Indexer: Wendy Catalano
- Interior Designer: David Futato
- Cover Designer: Ellie Volkhausen
- Illustrator: Rebecca Demarest
Revision History for the First Edition
- 2015-06-24: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491900987 for release details.
The OReilly logo is a registered trademark of OReilly Media, Inc. Hadoop Security, the cover image, and related trade dress are trademarks of OReilly Media, Inc.
While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
978-1-491-90098-7
[LSI]
Foreword
It has not been very long since the phrase Hadoop security was an oxymoron. Early versions of the big data platform, built and used at web companies like Yahoo! and Facebook, didnt try very hard to protect the data they stored. They didnt really have tovery little sensitive data went into Hadoop. Status updates and news stories arent attractive targets for bad guys. You dont have to work that hard to lock them down.
As the platform has moved into more traditional enterprise use, though, it has begun to work with more traditional enterprise data. Financial transactions, personal bank account and tax information, medical records, and similar kinds of data are exactly what bad guys are after. Because Hadoop is now used in retail, banking, and healthcare applications, it has attracted the attention of thieves as well.
And if data is a juicy target, big data may be the biggest and juiciest of all. Hadoop collects more data from more places, and combines and analyzes it in more ways than any predecessor system, ever. It creates tremendous value in doing so.
Clearly, then, Hadoop security is a big deal.
This book, written by two of the people whove been instrumental in driving security into the platform, tells the story of Hadoops evolution from its early, wide open consumer Internet days to its current status as a trusted place for sensitive data. Ben and Joey review the history of Hadoop security, covering its advances and its evolution alongside new business problems. They cover topics like identity, encryption, key management and business practices, and discuss them in a real-world context.
Its an interesting story. Hadoop today has come a long way from the software that Facebook chose for image storage a decade ago. It offers much more power, many more ways to process and analyze data, much more scale, and much better performance. Therefore it has more pieces that need to be secured, separately and in combination.
The best thing about this book, though, is that it doesnt merely describe. It prescribes. It tells you, very clearly and with the detail that you expect from seasoned practitioners who have built Hadoop and used it, how to manage your big data securely. It gives you the very best advice available on how to analyze, process, and understand data using the state-of-the-art platformand how to do so safely.
Mike Olson,
Chief Strategy Officer and Cofounder, Cloudera, Inc.
Preface
Apache Hadoop is still a relatively young technology, but that has not limited its rapid adoption and the explosion of tools that make up the vast ecosystem around it. This is certainly an exciting time for Hadoop users. While the opportunity to add value to an organization has never been greater, Hadoop still provides a lot of challenges to those responsible for securing access to data and ensuring that systems respect relevant policies and regulations. There exists a wealth of information available to developers building solutions with Hadoop and administrators seeking to deploy and operate it. However, guidance on how to design and implement a secure Hadoop deployment has been lacking.
This book provides in-depth information about the many security features available in Hadoop and organizes it using common computer security concepts. It begins with introductory material in the first chapter, followed by material organized into four larger parts: . These parts cover the early stages of designing a physical and logical security architecture all the way through implementing common security access controls and protecting data. Finally, the book wraps up with use cases that gather many of the concepts covered in the book into real-world examples.
Audience
This book targets Hadoop administrators charged with securing their big data platform and established security architects who need to design and integrate a Hadoop security plan within a larger enterprise architecture. It presents many Hadoop security concepts including authentication, authorization, accounting, encryption, and system architecture.
. Practiced security architects might want to skip that chapter unless theyre looking for a review. In general, we dont assume that you have a programming background, and try to focus on the architectural and operational aspects of implementing Hadoop security.
Conventions Used in This Book
The following typographical conventions are used in this book:
ItalicIndicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values determined by context.
Tip
This element signifies a tip or suggestion.
Note
This element signifies a general note.
Warning
This element indicates a warning or caution.
Using Code Examples
Throughout this book, we provide examples of configuration files to help guide you in securing your own Hadoop environment. A downloadable version of some of those examples is available at .
This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless youre reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from OReilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your products documentation does require permission.