Akidau Tyler - Streaming systems: the what, where, when, and how of large-scale data processing

by Tyler Akidau , Slava Chernyak , and Reuven Lax

Copyright 2018 Tyler Akidau, Slava Chernyak, and Reuven Lax. All rights reserved.

Printed in the United States of America.

Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

• Editors: Rachel Roumeliotis and Jeff Bleiel
• Production Editor: Nicholas Adams
• Copyeditor: Octal Publishing, Inc.
• Proofreader: Kim Cofer
• Indexer: Ellen Troutman-Zaig
• Interior Designer: David Futato
• Cover Designer: Karen Montgomery
• Illustrator: Rebecca Demarest

• August 2018: First Edition

See http://oreilly.com/catalog/errata.csp?isbn=9781491983874 for release details.

The OReilly logo is a registered trademark of OReilly Media, Inc. Streaming Systems, the cover image, and related trade dress are trademarks of OReilly Media, Inc.

While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-491-98387-4

[LSI]

Hello adventurous reader, welcome to our book! At this point, I assume that youre either interested in learning more about the wonders of stream processing or hoping to spend a few hours reading about the glory of the majestic brown trout. Either way, I salute you! That said, those of you in the latter bucket who dont also have an advanced understanding of computer science should consider how prepared you are to deal with disappointment before forging ahead; caveat piscator, and all that.

To set the tone for this book from the get go, I wanted to give you a heads up about a couple of things. First, this book is a little strange in that we have multiple authors, but were not pretending that we somehow all speak and write in the same voice like were weird identical triplets who happened to be born to different sets of parents. Because as interesting as that sounds, the end result would actually be less enjoyable to read. Instead, weve opted to each write in our own voices, and weve granted the book just enough self-awareness to be able to make reference to each of us where appropriate, but not so much self-awareness that it resents us for making it only into a book and not something cooler like a robot dinosaur with a Scottish accent.

As far as voices go, there are three youll come across:

That would be me. If you havent explicitly been told someone else is speaking, you can assume that its me, because we added the other authors somewhat late in the game, and I was basically like, hells no when I thought about going back and updating everything Id already written. Im the technical lead for the Data Processing Languages ands Systems group at Google, responsible for Google Cloud Dataflow, Googles Apache Beam efforts, as well as Google-internal data processing systems such as Flume, MillWheel, and MapReduce. Im also a founding Apache Beam PMC member.

Figure P-1. The cover that could have been...

Slava was a long-time member of the MillWheel team at Google, and later an original member of the Windmill team that built MillWheels successor, the heretofore unnamed system that powers the Streaming Engine in Google Cloud Dataflow. Slava is the foremost expert on watermarks and time semantics in stream processing systems the world over, period. You might find it unsurprising then that hes the author of .

Reuven is at the bottom of this list because he has more experience with stream processing than both Slava and me combined and would thus crush us if he were placed any higher. Reuven has created or led the creation of nearly all of the interesting systems-level magic in Googles general-purpose stream processing engines, including applying an untold amount of attention to detail in providing high-throughput, low-latency, exactly-once semantics in a system that nevertheless utilizes fine-grained checkpointing. You might find it unsurprising that hes the author of . He also happens to be an Apache Beam PMC member.