Danil Zburivsky - Designing Cloud Data Platforms

Danil Zburivsky and Lynda Partner

To comment go to liveBook

Manning

Shelter Island

For more information on this and other Manning titles go to

www.manning.com

For online information and ordering of these and other Manning books, please visit www.manning.com. The publisher offers discounts on these books when ordered in quantity.

For more information, please contact

Special Sales Department

Manning Publications Co.

20 Baldwin Road

PO Box 761

Shelter Island, NY 11964

Email: orders@manning.com

2021 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Mannings policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

ISBN: 9781617296444

This book is a true collaboration, a team effort between two very different people who share a love of data, new technology, and solving customer problems. We (Danil and Lynda) worked together at a data, analytics, and cloud IT services company for five years, where we partnered up to develop a cloud analytics practice. Danil, with his years of Hadoop experience, brought the technical chops, and Lynda brought the business perspective. We realized early on that both were needed to solve real-world data problems, and over time, Danil became more business-oriented and Lynda became knowledgeable enough about the cloud and data to contribute and even sometimes challenge Danil.

The move from Hadoop as a big data platform to cloud-native platforms for data and analytics was an easy onewe both love the promise of cloud and big data. With the support of our employer, we built an internal team and designed and delivered not just awesome technology solutions but solutions that delivered real business outcomes using data and cloud. We did this for dozens of customers, and over time, we developed a set of best practices and knowledge. It was this experience and our unique mix of technical and business skills that let us believe that we could take a really complicated technical subject and make it understandable for a broader audience. We started with blog posts and white papers, and when Manning called and asked if Danil wanted to write another book (his first was on Hadoop), it seemed right and natural to do it together.

Both of us were active speakers at industry events, so we took advantage of these opportunities to frame our ideas for the book and used audience feedback to refine them. We also agreed that we would weave in real customer stories because we both believe that stories make all learning easier. Once we realized that we were aligned on how to approach the book, there was nothing left but to start typing. It took almost two years, but we are both really happy with the outcome, and we hope you are too.

We knew this would be a lot of workDanil especially, because he had done it beforebut we both agree that it ended up being more work than either of us thought. We realized that we are both perfectionists, and we were each happy pushing the other to do better. The end result is a product we are both proud of, but we wouldnt be here without a broader team of people who supported us. Wed like to thank a few of them here.

First and foremost, we thank our spouses for putting up with our absences on weekends and holidays while we typed and typed and typed. It is still amazing to us that neither ever complained and both were always there for us.

Our work communities have been incredibly supportive starting with our employer at the time, Pythianespecially founder Paul Valle, who backed us when we said we thought we could develop and sell cloud-native data platforms at a time when people were saying cloud native what? Pythian also graciously allowed us to use the diagrams that appear in chapter 10. The bottom line is that our employers, past and present, have encouraged us to keep writing and sharing our knowledge, and we are grateful for that support.

A big, big thank you goes to the Kick AaaS team we worked withKevin Pedersen, Christos Soulios, Valentin Nikotin, and Rory Bramwell, who all took a gamble on a new direction and followed us into the unknownthey are the invisible authors of this book. And we will never be able to thank our customers enough, especially the first few who were patient as we learned how to make better and better designs on their behalf.

Next, wed like to acknowledge the folks at Manning, especially our editor Susan Ethridge, who seemed to know just how hard to push to get us to produce our best work but who also knew when we just needed a sympathetic ear. The book is better because of you, Susan, and youve become a friend in the process. We will miss our weekly meetings! Deirdre Hiam, our project editor; Katie Petito, our copyeditor; Katie Tennant, our proofreader; and Mihaela Batinic, our reviewing editor: thank you, all.

And lastly, we thank the people who reviewed our book as we wrote it and gave great constructive feedback. Were not going to lie, it was really difficult to send out our opus to strangers and ask them for feedback, but we did and we got great suggestions back. Thank you for taking the time, for being so constructive, and for the nice things you said about the bookit kept us going through the hard times. Thank you, all the reviewers whose suggestions helped make this a better book: Robert Wenner first and foremost (What will Robert say? became a common refrain in our meetings), Alain Couniot, Alex Saez, Borko Djurkovic, Chris Viner, Christopher E. Phillips, Daniel Berecz, Dave Corun, David Allen Blubaugh, David Krief, Emanuele Piccinelli, Eros Pedrini, Gabor Gollnhofer, George Thomas, Hugo Cruz, Jason Rendel, Ken Fricklas, MikeJensen, Peter Bishop, Peter Hampton, Richard Vaughan, Sambasiva Andaluri, Satej Sahu, Sean Thomas Booker, Simone Sguazza, Ubaldo Pescatore, and Vishwesh Ravi Shrimali.