Mahmoud Parsian - Data Algorithms with Spark: Recipes and Design Patterns for Scaling Up using PySpark

• 2020-08-26: First Release
• 2021-01-20: Second Release
• 2021-05-24: Third Release
• 2021-09-10: Fourth Release

With Early Release ebooks, you get books in their earliest formthe authors raw and unedited content as they writeso you can take advantage of these technologies long before the official release of these titles.

This will be the 1st chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at mpotter@oreilly.com.

Spark is a powerful analytics enginefor large-scale data processing.Spark aims at light speed, ease of use,extensibility in big data. Spark hasintroduced high-level APIs in Java,Scala, Python, and R. In this book,I use PySpark (Python API for Spark)for the simplicity.

The purpose of this chapter is to introducePySpark as a main component of Spark echosystem and show that it can effectively beused to solve big data problems such as ETL,indexing billions of documents, ingestingmillions of genomes, machine learningalgorithms, graph data analysis, DNA dataanalysis, and much more. To understandPySpark , we will review Spark and PySparkarchitectures and provide examples to showthe expressive power of PySpark. The reasonI chose Spark is very simple: Spark is a provenand well-adopted technology at many hightech industries, who handle big data everyday. Spark is an analytics engine designedfor large-scale distributed data processing,in the cloud or private data centers.

This chapter covers

• Why Spark for data analytics?

• What is PySpark API?

• What problems does PySpark solve?

• PySpark with examples

• Interactive PySpark Shell

• Running a PySpark application

This book provides a practical introductionto data algorithms (practical and pragmaticbig data transformations) using PySpark (PythonAPI for Spark). To get the most out of thisbook, you should be comfortable reading andwriting very basic Python code, and have somefamiliarity with fundamentals of algorithms(how to solve computational problems in simplesteps). By data algorithms, I mean solving bigdata problems by using a set of scalable andefficient data transformations (such as mappers,reducers, filters, ).

In this book, I explore and teach the followingbasic concepts:

• How to design and develop data algorithmsby using PySpark

• How to solve big data problems by usingSparks data abstractions (RDDs, DataFrames,and GraphFrames)

• How to partition big data in such a wayto achieve maximum performance of ETLand data transformations such as summarydata patterns

• How to integrate fundamental data designpatterns (such as Top-10, MinMax, InMapperCombiners, ) with a useful and practicalSpark transformations by using PySpark

In this book, I will provide simple workingexamples in PySpark so that you can adapt(cut-and-past-and-modify) them to solveyour big data problems. All of the sourcecode in this book is hosted at the GitHub Data Algorithms with Spark.

My focus in this book is on PySpark ratherthan Spark. This means that the providedexamples will focus on solving big dataproblems using PySpark and on getting effectiveby using PySpark and not an exhaustive referenceon Sparks features.

This chapter will present an introductionto Spark, PySpark, and data analytics withPySpark. We will present the corearchitecture of Spark, PySpark, SparkApplication, and Sparks Structured APIsusing RDDs (Resilient DistributedDataset), DataFrames, and SQL. We will touchon Sparks core functions (transformationsand actions) and concepts so that you areempowered to start using Spark and PySparkright away. Sparks data abstractions are RDDs,DataFrames, and DataSets. This means that youcan represent your data (stored as Hadoop files,Amazon S3 objects, Linux files, collection datastructures, relational database tables, )in any combinations of RDDs and DataFrames.

Once your data is represented in a Spark dataabstraction (such as an RDD or a DataFrame),then you may apply transformations on them andcreate new data abstractions until you come upwith final desired data. Sparks transformations(such as map() and reduceBykey()) can be usedto convert your data from one form into anotherone until you get your desired result. I willexplain these data abstractions shortly.

Source Code

Note

Complete programs for this chapter are presented inGitHub Chapter 1.

Why Spark for Data Analytics

Why should we use Spark? Spark is apowerful analytics engine for largescale data processing. The most importantreasons for using Spark are listed:

• Spark is simple, powerful, and fast(uses RAM rather than diskSparkruns workloads 100x faster.)

• Spark is open-source, free, and cansolve any big data problem

• Spark runs everywhere (Hadoop, Mesos,Kubernetes, standalone, or in the cloud).

• Spark can read/write data from/to manydata sources

• Spark can read/write data in row-basedand column-based (such as Parquet andORC) formats

Sparks high level architecture is illustratedby Figure 1.1.