Josh Fischer - Grokking Streaming Systems: Real-time event processing

Real-time event processing

Josh Fischer and Ning Wang

To comment go to liveBook

Manning

Shelter Island

For more information on this and other Manning titles go to

www.manning.com

For online information and ordering of these and other Manning books, please visit www.manning.com. The publisher offers discounts on these books when ordered in quantity.

For more information, please contact

Special Sales Department

Manning Publications Co.

20 Baldwin Road

PO Box 761

Shelter Island, NY 11964

Email: orders@manning.com

2022 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

ISBN: 9781617297304

A mentor of mine once told me, at the beginning of my tech career, If theres one thing you can do to better your career, its contributing to open source. Id harbored that thought in the back of my mind throughout the years but never had a reason to do so. I thought, What could I build that would be useful for others? While working at 1904labs I developed the ECO API for (at the time) Twitter Heron. It came from a clients needand from a little bit of selfishness; I really wanted to write and contribute that code. Eventually, Twitter donated Heron to the Apache Foundation, and I was invited to be a committer and part of the project management committee for Heron. The project interested me because it was the first open source project I did a deep dive on.

About a year later, from that initial commit on Herons main branch at about 4 p.m. on a Monday, I received an email with the subject line, Apache Heron Book or Course Project from Eleonor Gardner. After a quick read, I almost discarded the email, thinking it was a hoax. After all, why would anyone want me to write a book or teach a course project? Well, how wrong was I? After a discussion with Mike Stephens, Mannings associate publisher, and a few email exchanges with his assistant, Eleonor, I knew I needed some help. I reached out to my friend and fellow Apache Heron committer, Ning Wang, praying that hed be interested in writing a book with me. Luckily, he wasand that was the start to our long and rewarding journey.

Initially, the conversations about this book were for us to write specifically about Heron. But Ning had some ideas to make the book better. After all, technologies change quickly and breaking changes in software can make a book obsolete quickly. We wanted to write about a topic that would live beyond individual streaming frameworks. We agreed to write a framework-agnostic book to teach the core concepts in a way that would allow readers to be able to jump into any streaming frameworks documentation and hit the ground running.

So, we started writing the book using only words and then Ning and I were gently guided to try another approach. Again. And again. And again. And again. We learned that diagrams make the content of a book much easier for readers to absorb. We created our first diagrams on paper with pen, and they were dismal:

Over the course of writing the book, our primitive-looking, scrawled creations evolved into the diagrams you now see in the book. Ning and I designed and developed all of these diagrams ourselves. We are extremely proud of what we have created, and we hope that you see value in this book.

Josh Fischer, November 2021

First, I must thank my kids and my ever-so-wonderful partner, Melissa. She is the most patient and fabulous person anyone could ever ask for. She has helped me endure all the tough spots of life while writing this book. My kidsAiden, Wes, Hollyn, Oliver, Declan, and Dylanhave been patient, and often self-entertaining, on the late nights or early mornings while I took time to write.

Thank you, Ning, for sticking with me through the process of writing. Learning from you has been one of the greatest benefits of writing this book.

I must thank Dan Tumminello, Dave Lodes, Laura Stobie, Jim Towey, Steve Willis, Mike Banocy, Sean Walsh, Pavan Veeramachineni, Robert McMillan, Chad Storm, Karthik Ramasamy, and Chandra Shekar. All of them have been a great influence on me personally and professionally.

Last but not least, I want to thank Bert Bates. He is without a doubt the most patient, forgiving, and all-around fantastic teacher I have ever had. Becky Whitney always participated in conversations that may have been tough, but kept us on track to deliver for Manning. Thank you, Mike Stephens, for giving me a chance. Eleonor Gardner set up our initial conversations, and, finally, Andy Marinkovich and Keri Hales, who put the finishing touches on the book.

To all the reviewers, Andres Sacco, Anto Aravinth, Anupam Sengupta, Apoorv Gupta, Beau Bender, Brent Honadel, Brynjar Smri Bjarnason, Chris Lundberg, Cicero Zandona, Damian Esteban, Deepika Fernandez, Fernando Antonio da Silva Bernardino, Johannes Lochmann, Kent R. Spillner, Kumar Unnikrishnan, Lev Andelman, Marc Roulleau, Massimo Siani, Matthias Busch, Miguel Montalvo, Sebastin Palma, Simeon Leyzerzon, Simon Seyag, and Simon Verhoeven: your comments, questions, and concerns have all made this a better book. Thank you.

Josh Fischer, November 2021

Two years! I have lost count of how many people I need to thank. This book wouldnt be possible without any of the people listed here, as well as many others not listed.