Rukmani Gopalan - The Cloud Data Lake: A Guide to Building Robust Cloud Data Architecture. Early Release

• 2022-05-03: First Release
• 2022-06-16: Second Release
• 2022-07-15: Third Release
• 2022-08-18: Fourth Release

With Early Release ebooks, you get books in their earliest formthe authors raw and unedited content as they writeso you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at .

Without big data, you are blind and deaf and in the middle of a freeway.

Geoffrey Moore

If we were playing workplace Bingo, there is a high chance you would win a full house by crossing off all these words that you have heard in your organization in the past 3 months - digital transformation, data strategy, transformational insights, data lake, warehouse, data science, machine learning, and intelligence. It is now common knowledge that data is a key ingredient for organizations to succeed, and organizations that rely on data and AI clearly outperform their contenders. According to an IDC study sponsored by Seagate, the amount of data that is captured, collected, or replicated is expected to grow to 175 ZB by the year 2025. This data that captured, collected, or replicated is referred to as the Global Datasphere. This data comes from three classes of sources :

• The core - traditional or cloud based datacenters.

• The edge - hardened infrastructure, such as the cell towers.

• The endpoints - PC, tablets, smartphones, and IoT devices.

This study also predicts that 49% of this Global Datasphere will be residing in public cloud environments by the year 2025.

If you have ever wondered, Why does this data need to be stored? What is it good for?, the answer is very simple - think of all of these data available as bits and pieces of words strewn around the globe in different languages and scripts, each sharing a sliver of information, like a piece in a puzzle. Stitching them together in a meaningful fashion tells a story that not only informs, but also could transform businesses, people, and even how this world runs. Most successful organizations already leverage data to understand the growth drivers for their businesses and the perceived customer experiences and taking the rightful action - looking at the funnel or customer acquisition, adoption, engagement, and retention are now largely the lingua franca of funding product investments. These types of data processing and analysis are referred to as business intelligence, or BI, and are classified as offline insights. Essentially, the data and the insights are crucial in presenting the trend that shows growth so the business leaders can take action, however, this workstream is separate to the core business logic that is used to run the business itself. As the maturity of the data platform grows, an inevitable signal we get from all custoemrs is that they start getting more requests to run more scenarios on their data lake, truly adhering to the Data is the new oil idiom.

Organizations leverage data to understand the growth drivers for their business and the perceived customer experience. They can then leverage data to set targets and drive improvements in customer experience with better support and newer features, they can additionally create better marketing strategies to grow their business and also drive efficiencies to lower their cost of building their products and organizations. Starbucks, the coffee shop that is present around the globe, uses data in every place possible to continously measure and improve their business. They use the data from their mobile applications and correlate that with their ordering system to better understand customer usage patterns and send targeted marketing campaigns. They use sensors on their coffee machines that emit health data every few seconds, and this data is analyzed to drive improvements into their predictive maintenance, they also use these connected coffee machines to download recipes to their coffee machines without involving human intervention. As the world is just learning to cope with the pandemic, organizations are leveraging data heavily to not just transform their businesses, but also to measure the health and productivity of their organizations to help their employees feel connected and minimize burn out. Overall, data is also used for world saving initiatives such as Project Zamba that leverages artificial intelligence for wildlife research and conservation in the remote jungles of Africa, and leveraging IoT and data science to create a circular economy to promote environmental sustaintability.