Rukmani Gopalan - The Cloud Data Lake

by Rukmani Gopalan

Copyright 2022 Rukmani Gopalan. All rights reserved.

Printed in the United States of America.

Published by OReilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.

OReilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com .

• Editors: Andy Kwan and Jill Leonard
• Production Editor: Ashley Stussy
• Interior Designer: David Futato
• Cover Designer: Karen Montgomery
• Illustrator: Kate Dullea

• March 2023: First Edition

See http://oreilly.com/catalog/errata.csp?isbn=9781098116583 for release details.

The OReilly logo is a registered trademark of OReilly Media, Inc. The Cloud Data Lake, the cover image, and related trade dress are trademarks of OReilly Media, Inc.

The views expressed in this work are those of the author(s), and do not represent the publishers views. While the publisher and the author(s) have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author(s) disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

978-1-098-11652-1

[FILL IN]

Its 6 in the morning, your phone gently wakes you up, and automatically turns on your notifications. Your smart refridgerator reminds you that you need to order milk, and shows you an option to place an order to buy more milk since it knows you are running low. You do that and hop on your exercise machine, where you see personalized picks for you based on your workout routines. You get ready and eat breakfast without bothering to look at the clock because you know your phone will tell you when it is time to start driving based on what it has learnt about your commute and the traffic patterns. As you leave the door, your smart home assistant ensures the lights are turned off and the doors are locked. What would have seemed like science fiction a few decades ago is a regular day in our lives now. All this is made possible by the leaps technology has made in three key areas - devices that have made computing ubiqutous, connectivity that has shrunk the world by bringing the knowledge of the internet to these devices, and the technology (data/artificial intelligence/machine learning) that has helped devices learn patterns and make decisions. Data is now at the heart of how the world operates and organizations are increasingly relying on data to both inform and transform their business.

My mind goes back to 2013, when my own personal journey with data started as I worked on identity and personalization services for Microsoft Office. It was a year of great many learnings for me. I understood what it meant to develop cloud based applications, inclduing the nuances of building a direct to consumer experience vs an enterprise ready application. Most of all though, I was thrilled at the possibility of having a direct connection to customer experiences from these cloud services. When we shipped boxed products, i.e. products that shipped in a CD or DVD, and had customers install them on their devices, the only way for us to understand their experiences was to get anonymized telemetry data, organized user research studies or focus groups, or when the customer has issues through support cases. A lot of our insights on the product usage were based on the data from the customers who opted to talk to us, which was a minute fraction. With the cloud services I built, I had a real time understanding of my customers. This helped us tune our services and deliver more personalized experiences to our users. We were able to experiment with variations of features with our customers to better understand what helps more with their productivity. Since then, I have been working on various platforms and cloud services and I realize how the value of the data, when amplified with the elasticity of the cloud, can help inform and transform businesses.