Lad Sagar - Mastering Databricks Lakehouse Platform

My sisters Mohini Lad and Jigna Mistry whose
constant warmth and companionship I couldnt repay
even with a lifetime of dog treats

Sagar Lad is a Technical Solution Architect with a leading multinational software company and has deep expertise in implementing Data & Analytics solutions for large enterprises using Cloud and Artificial Intelligence. He is an experienced Azure Platform evangelist with strong focus on driving cloud adoption for enterprise organizations using Microsoft Cloud Solutions & Offerings with 8+ Years of IT experience. He loves blogging and is an active blogger on medium,linkedin and C# Corner developer community. He was awarded the C# Corner MVP in September 2021, for his contributions to the developer community.

Certifications to his credit https://www.credly.com/users/sagar-lad

Twitter: @AzureSagar( https://twitter.com/AzureSagar)

Medium: https://medium.com/@sagu94271

Anjani Kumar is the MD and Founder of MultiCloud4u which is one of the fastest DIgital transformation startups extensively using Data Driven solutions.

As a technologist Anjani is a multifaceted Enterprise and Data Solution Architect and have consulted 100 of top fortune 500 clients for setting up large data warehouse and digital business transformations correctly while working with publicis.Sapient, Royal Bank Of Scotland, Microsoft, AMEX and multiple other clients such as Unilever, Citi Bank, Brown Advisory, Nissan, Sprint, Bata, Philips, Jera Americas and many more.He runs a global knowledge sharing platform called 5thir www.5thir.com

LinkedIn Profile:

https://www.linkedin.com/in/anjanikkumar/

Ilse Epskamp is an IT Engineer with 7+ years of IT experience in the banking industry. With a strong focus on automation, she designs and develops sustainable data solutions on the Azure platform. Ilse is experienced in building automated workflows for loading, validating, transforming and ingesting data using various Azure Offerings and programming languages such as Python, as well as handling CI/CD deployment and code management with Azure DevOps. She has an educational background in the field of Artificial Intelligence and Data Science Management, is Azure certified, and enjoys expanding her technical knowledge by trainings and hands-on projects. She is a mother of two girls, living in Alkmaar, The Netherlands.

LinkedIn:

https://www.linkedin.com/in/ilse-epskamp

I would like to thank the entire BPB team for giving us this opportunity to author this book and guiding throughout the journey, and trusting us with this assignment.

I would like to thank the editing team for editing this book and making the manuscript look neat and perfect.

You build your data platform and then configure the databases to build data infrasture and data pipelines. Setting up the data pipeline or ETL pipeline is time-consuming. We then use various tools like Informatica etc to build ETL pipelines. All these activities take a lot of time and effort. You may also end up with issues while setting up the environment and may have to spend much effort to fix these issues, which is in no way related to your application. You must ensure that the data pipelines are up and running always and fix all hardware and hosting environment software failures that may bring the hosted application down. To sum up, you end up with a huge amount of capital and operational expenditure and effort.

Alternatively, you can build your data solutions using databricks lakehouse platform which is more stable,efficient and cheaper. You then create and schedule data pipelines with a click of button and pricing will be pay as per use. This approach would save you hardware capital cost and operational cost to some extent.

When we use Databicks Lakehouse Platform,underlying platform manages the configurations and administration task. You can focus on building your data and AI solutions using the Databricks Lakehouse platform with integration to Azure Data Factory,Power BI, Tableau etc. It has in-built feature of high availability, reliability, fault tolerance, scaling, and many more for this services.

The book starts with basic concepts of databricks lakehouse platform. We get an understanding of the real-world problems to implement data and analytics solution using Delta Lake. Then we explore about change data feed offerings and take a look at all necessary Storage Services required to store data. We get an understanding of Azure Data Factory,Databricks and its benefits. We then deep dive into detailed Architecture and devops ci/cd implementation for your data platform.

DevOps is an essential aspect of any application development process. The next chapter deals with how to create Continuous Integration and Continuous Delivery pipelines using Azure DevOps. Here we get to learn how to create generic one click pipeline to deploy your data solutions to multiple environments.