Jan Lukavský - Building Big Data Pipelines with Apache Beam: Use a single programming model for both batch and stream data processing

Copyright 2021 Packt Publishing

This is an Early Access product. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the content and extracts of this book may evolve as it is being developed to ensure it is up-to-date.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, ortransmitted in any form or by any means, without the prior written permission of the publisher,except in the case of brief quotations embedded in critical articles or reviews.

The information contained in this book is sold without warranty, either express or implied. Neither the author nor Packt Publishing or its dealers and distributors will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Early Access Publication: Building Big Data Pipelines with Apache Beam

Early Access Production Reference: B16918

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK

ISBN: 978-1-80056-493-0

Welcome to Packt Early Access. Were giving you an exclusive preview of this book before it goes on sale. It can take many months to write a book, but our authors have cutting-edge information to share with you today. Early Access gives you an insight into the latest developments by making chapter drafts available. The chapters may be a little rough around the edges right now, but our authors will update them over time. Youll be notified when a new version is ready.

This title is in development, with more chapters still to be written, which means you have the opportunity to have your say about the content. We want to publish books that provide useful information to you and other customers, so well send questionnaires out to you regularly. All feedback is helpful, so please be open about your thoughts and opinions. Our editors will work their magic on the text of the book, so wed like your input on the technical elements and your experience as a reader. Well also provide frequent updates on how our authors have changed their chapters based on your feedback.

You can dip in and out of this book or follow along from start to finish; Early Access is designed to be flexible. We hope you enjoy getting to know more about the process of writing a Packt book. Join the exploration of new topics by contributing your ideas and see them come to life in print.

1. Introduction to Data Processing with Apache Beam
2. Implementing, Testing, and Deploying Basic Pipelines
3. Implementing Pipelines using Stateful Processing
4. Structuring Code for Reusability
5. Using SQL for Pipeline Implementation
6. Using Your Preferred Language with Portability
7. Extending Apache Beam's IO Connectors
8. Understanding How Runners Execute Pipelines

Data. Big Data. Real-time Data. Data Streams. Many buzzwords to describe many things, and yet they have many common properties. Many mind-blowing applications arise from successful application of theoretically simple logic take data and produce knowledge. But even such a simple-sounding task turns out to be difficult when the amount of data needed to produce knowledge is ... huge, and still growing. Given the vast volumes of data produced by humanity every day, which tools to choose to turn our simple logic into scalable solutions? Into solutions that protect our investment into creating the data extraction logic, even in the presence of new requirements arising, or changing on a daily basis, and new data processing technologies being created? This book focuses on describing why Apache Beam might be a good solution to these challenges and provides a guide through the learning process.

In this chapter, we will cover the following topics:

• Why Apache Beam?
• Writing your first Pipeline
• Running a Pipeline against streaming data
• Measuring event time progress inside data streams
• Assigning data to windows
• Defining the lifecycle of a state in terms of windows
• Unifying batch and streaming data processing

In this first chapter we will introduce some elementary Pipelines written using Apache Beams Java SDK. We will use the code located on GitHub, at: https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam

We will need the following tools:

• JDK11, possibly OpenJDK11 installed, JAVA_HOME set appropriately
• git
• Bash

Important note

Although it is possible to run many tools in this book using Windows shell, throughout this book we will focus on using Bash scripting only. We hope Windows users will be able to run Bash using virtualization or The Windows Subsystem for Linux or any similar technology.

First of all, we need to clone the repository.

1. We create a suitable directory for that and run:

   $ git clone https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam.git

2. This will result in a directory Building-Big-Data-Pipelines-with-Apache-Beam being create in the working directory. We then run in this newly created directory:

   $ ./mvnw clean install

Throughout this book, we will use the notation, that the character $ denotes a Bash shell., therefore $ ./mvnw clean install would mean to run command ./mvnw in the top-level directory of the git clone (named Building-Big-Data-Pipelines-with-Apache-Beam). By using chapter1$ ../mvnw clean install we mean to run the specified command in subdirectory called chapter1.

There are several basic questions one might ask when considering a new technology to learn and apply in practice. Two fundamental questions might be:

• What problem that Im struggling with can the new technology help me solve?
• What would be the costs?

Each sound technology has a well-defined selling point, something that can justify its existence in the presence of other competitive technologies. In the case of Apache Beam, this selling point could be reduced to single word: portability. Apache Beam is portable on several layers, namely:

• Portability of Apache Beam Pipelines between multiple Runners (by Runner we mean a technology, that actually executes distributed computation described by a Pipelines author).
• Portability of Data Processing Model of Apache Beam to various programming languages.
• Portability of Data Processing logic between Bounded and Unbounded Data.

Each of these points deserves a few words of explanation. By Runner portability, we mean the possibility to run existing Pipelines written in one of the supported programming languages (for instance Java, Python, Go, Scala or even SQL) against a data processing engine that can be chosen