Sev Leonard - Cost-Effective Data Pipelines

• 2022-09-21: First Release
• 2022-12-08: Second Release
• 2023-01-24: Third Release
• 2023-03-15: Fourth Release

With Early Release ebooks, you get books in their earliest formthe authors raw and unedited content as they writeso you can take advantage of these technologies long before the official release of these titles.

This will be the Preface of the final book. Please note that the GitHub repo is available here: https://github.com/gizm00/oreilly_dataeng_book.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at vwilson@oreilly.com.

In my work on data pipelines, the biggest cost impact I have seen to date was due to a bug. For months our pipeline was incorrectly transforming data, undetected until our customers noticed the data was wrong.

We could have caught this bug with schema validation tests, which youll learn about in this book. Instead, we spent a significant chunk of our annual cloud bill recomputing the bad data. It cost us the trust of our customers as well, to the point that the validity of the project as a whole was questioned. Sure, the cloud bill was bad, but the costs of inadequate development practices nearly scuttling the project was worse.

If you search the web for cloud cost optimization strategies you may read horror stories about an Amazon Web Services (AWS) Lambda function gone awry, or get some vague advice that you should right size your compute resources without any specifics on how to do it. These are important strategies that you will learn how to do in this book, but theres more to it than that.

My experience has been that costs in cloud data pipeline development come from the difficulties of wrangling a system that spans unknown third party data sources, cloud services, extremely sophisticated big data processing engines, and multiple code bases. Couple this with a fast-paced production environment and you can quickly devolve into a reactive work mode where code turns into spaghetti, pipelines become difficult to evolve and test, and no one knows whats really going on because theres insufficient monitoring.

Altogether this can create an environment where change is hard, resulting in longer lead times for bringing new functionality onboard. Bugs and burnout are common, eroding customer trust and adoption. These issues hit a companys bottom line in more ways than just the cloud bill.

I wrote this book because this isnt the way it has to be. With a focus on effective monitoring, software development best practices, and targeted advice on designing cloud compute and storage, this book will get you set up for success from the outset and enable you to manage the evolution of data pipelines in a cost effective way.

Ive used these approaches in batch and streaming systems, handling anywhere from a few thousand rows to petabytes of data running the gambit from well-defined, structured data to semi-structured sources that change frequently.