Nathan Marz - Big Data: Principles and best practices of scalable realtime data systems

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

Special Sales DepartmentManning Publications Co.20 Baldwin RoadPO Box 761Shelter Island, NY 11964Email: orders@manning.com

2015 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

Manning Publications Co.20 Baldwin RoadPO Box 761Shelter Island, NY 11964

Development editors: Renae Gregoire, Jennifer StoutTechnical development editor: Jerry GainesCopyeditor: Andy CarrollProofreader: Katie TennantTechnical proofreader: Jerry KuchTypesetter: Gordan SalinovicCover designer: Marija Tudor

ISBN 9781617290343

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 EBM 20 19 18 17 16 15

When I first entered the world of Big Data, it felt like the Wild West of software development. Many were abandoning the relational database and its familiar comforts for NoSQL databases with highly restricted data models designed to scale to thousands of machines. The number of NoSQL databases, many of them with only minor differences between them, became overwhelming. A new project called Hadoop began to make waves, promising the ability to do deep analyses on huge amounts of data. Making sense of how to use these new tools was bewildering.

At the time, I was trying to handle the scaling problems we were faced with at the company at which I worked. The architecture was intimidatingly complexa web of sharded relational databases, queues, workers, masters, and slaves. Corruption had worked its way into the databases, and special code existed in the application to handle the corruption. Slaves were always behind. I decided to explore alternative Big Data technologies to see if there was a better design for our data architecture.

One experience from my early software-engineering career deeply shaped my view of how systems should be architected. A coworker of mine had spent a few weeks collecting data from the internet onto a shared filesystem. He was waiting to collect enough data so that he could perform an analysis on it. One day while doing some routine maintenance, I accidentally deleted all of my coworkers data, setting him behind weeks on his project.

I knew I had made a big mistake, but as a new software engineer I didnt know what the consequences would be. Was I going to get fired for being so careless? I sent out an email to the team apologizing profuselyand to my great surprise, everyone was very sympathetic. Ill never forget when a coworker came to my desk, patted my back, and said Congratulations. Youre now a professional software engineer.

In his joking statement lay a deep unspoken truism in software development: we dont know how to make perfect software. Bugs can and do get deployed to production. If the application can write to the database, a bug can write to the database as well. When I set about redesigning our data architecture, this experience profoundly affected me. I knew our new architecture not only had to be scalable, tolerant to machine failure, and easy to reason aboutbut tolerant of human mistakes as well.

My experience re-architecting that system led me down a path that caused me to question everything I thought was true about databases and data management. I came up with an architecture based on immutable data and batch computation, and I was astonished by how much simpler the new system was compared to one based solely on incremental computation. Everything became easier, including operations, evolving the system to support new features, recovering from human mistakes, and doing performance optimization. The approach was so generic that it seemed like it could be used for any data system.

Something confused me though. When I looked at the rest of the industry, I saw that hardly anyone was using similar techniques. Instead, daunting amounts of complexity were embraced in the use of architectures based on huge clusters of incrementally updated databases. So many of the complexities in those architectures were either completely avoided or greatly softened by the approach I had developed.

Over the next few years, I expanded on the approach and formalized it into what I dubbed the Lambda Architecture. When working on a startup called BackType, our team of five built a social media analytics product that provided a diverse set of realtime analytics on over 100 TB of data. Our small team also managed deployment, operations, and monitoring of the system on a cluster of hundreds of machines. When we showed people our product, they were astonished that we were a team of only five people. They would often ask How can so few people do so much? My answer was simple: Its not what were doing, but what were not doing. By using the Lambda Architecture, we avoided the complexities that plague traditional architectures. By avoiding those complexities, we became dramatically more productive.

The Big Data movement has only magnified the complexities that have existed in data architectures for decades. Any architecture based primarily on large databases that are updated incrementally will suffer from these complexities, causing bugs, burdensome operations, and hampered productivity. Although SQL and NoSQL databases are often painted as opposites or as duals of each other, at a fundamental level they are really the same. They encourage this same architecture with its inevitable complexities. Complexity is a vicious beast, and it will bite you regardless of whether you acknowledge it or not.

This book is the result of my desire to spread the knowledge of the Lambda Architecture and how it avoids the complexities of traditional architectures. It is the book I wish I had when I started working with Big Data. I hope you treat this book as a journeya journey to challenge what you thought you knew about data systems, and to discover that working with Big Data can be elegant, simple, and fun.

N ATHAN M ARZ

This book would not have been possible without the help and support of so many individuals around the world. I must start with my parents, who instilled in me from a young age a love of learning and exploring the world around me. They always encouraged me in all my career pursuits.