Ed Elliott - Introducing .NET for Apache Spark: Distributed Processing for Massive Datasets

Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the books product page, located at www.apress.com/9781484269916 . For more detailed information, please visit http://www.apress.com/source-code .

From the first time I ever used Apache Spark, I was captivated by how powerful it was and what it could enable. Having come from a traditional Microsoft developer background, C, C++, C#, F#, Microsoft SQL Server, SSIS, and then onto Azure, Azure Data Factory, and SQL Data Warehouse, to be able to use Apache Spark to process large files quickly was fascinating. The less fascinating part was having to decide whether to learn Scala or Python to work with Apache Spark. I studied both languages but missed developing in C# and F#, so when at the launch of the .NET for Apache Spark project, I was particularly excited. I have been closely following the project and have submitted several successful pull requests, which are now included in the bindings.

This book intends to introduce both Apache Spark and the .NET for Apache Spark bindings. If you are new to Apache Spark but have C# or F# experience, or whether you already know Apache Spark but will use the .NET for Apache Spark bindings, then you will be able to use this book to get you started.

In this book, we will start with a step-by-step guide on getting an instance of Apache Spark running on your developer machine, whether on Windows, Linux, or macOS, including a discussion on the crucial points such as the Java version. We will cover in detail how the .NET bindings work. We will then get you up and running with your first .NET for Apache Spark application before explaining the different APIs and taking a look at some example programs.

We will walk through an example batch mode, streaming, and machine learning application that you can follow along. We will also look at how to troubleshoot Apache Spark and, finally, look at how you can make your changes to the .NET for Apache Spark project, including submitting any pull requests back to the core product, if you would like to. Finally, we will look at the support for the delta format, which brings ACID properties to files in a data lake. Exciting times!

Apache Spark is a data analytics platform that has made big data accessible and brings large-scale data processing into the reach of every developer. With Apache Spark, it is as easy to read from a single CSV file on your local machine as it is to read from a million CSV files in a data lake.

Let us look at an example . The code in Listings (the F# version) reads from a set of CSV files and counts how many records match a specific condition. The code reads all CSV files in a specific path, so the number of files we read from is practically limitless.

Although the examples in this chapter are fully functioning samples, they require a working Apache Spark instance, either locally or on a cluster. We cover setting up Apache Spark in Chapter .