Unknown - Frank Kanes Taming Big Data with Apache Spark and Python

• 1: Getting Started with Spark
  • b'Chapter 1: Getting Started with Spark'
  • b'Getting set up - installing Python, a JDK, and Spark and its dependencies'
  • b'Installing the MovieLens movie rating dataset'
  • b'Run your first Spark program - the ratings histogram example'
  • b'Summary'

• 2: Spark Basics and Spark Examples
  • b'Chapter 2: Spark Basics and Spark Examples'
  • b'What is Spark?'
  • b'The Resilient Distributed Dataset (RDD)'
  • b'Ratings histogram walk-through'
  • b'Key/value RDDs and the average friends by age example'
  • b'Running the average friends by age example'
  • b'Filtering RDDs and the minimum temperature by location example'
  • b'Running the minimum temperature example and modifying it for maximums'
  • b'Running the maximum temperature by location example'
  • b'Counting word occurrences using flatmap()'
  • b'Improving the word-count script with regular expressions'
  • b'Sorting the word count results'
  • b'Find the total amount spent by customer'
  • b'Check your results and sort them by the total amount spent'
  • b'Check your sorted implementation and results against mine'
  • b'Summary'

• 3: Advanced Examples of Spark Programs
  • b'Chapter 3: Advanced Examples of Spark Programs'
  • b'Finding the most popular movie'
  • b'Using broadcast variables to display movie names instead of ID numbers'
  • b'Finding the most popular superhero in a social graph'
  • b'Running the script - discover who the most popular superhero is'
  • b'Superhero degrees of separation - introducing the breadth-first search algorithm'
  • b'Accumulators and implementing BFS in Spark'
  • b'Superhero degrees of separation - review the code and run it'
  • b'Item-based collaborative filtering in Spark, cache(), and persist()'
  • b'Running the similar-movies script using Spark's cluster manager'
  • b'Improving the quality of the similar movies example'
  • b'Summary'

• 4: Running Spark on a Cluster
  • b'Chapter 4: Running Spark on a Cluster'
  • b'Introducing Elastic MapReduce'
  • b'Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY'
  • b'Partitioning'
  • b'Creating similar movies from one million ratings - part 1'
  • b'Creating similar movies from one million ratings - part 2'
  • b'Creating similar movies from one million ratings \xc3\xa2\xc2\x80\xc2\x93 part 3'
  • b'Troubleshooting Spark on a cluster'
  • b'More troubleshooting and managing dependencies'
  • b'Summary'

• 5: SparkSQL, DataFrames, and DataSets
  • b'Chapter 5: SparkSQL, DataFrames, and DataSets'
  • b'Introducing SparkSQL'
  • b'Executing SQL commands and SQL-style functions on a DataFrame'
  • b'Using DataFrames instead of RDDs'
  • b'Summary'

• 6: Other Spark Technologies and Libraries
  • b'Chapter 6: Other Spark Technologies and Libraries'
  • b'Introducing MLlib'
  • b'Using MLlib to produce movie recommendations'
  • b'Analyzing the ALS recommendations results'
  • b'Using DataFrames with MLlib'
  • b'Spark Streaming and GraphX'
  • b'Summary'

• 7: Where to Go From Here? Learning More About Spark and Data Science
  • b'Chapter 7: Where to Go From Here? \xe2\x80\x93 Learning More About Spark and Data Science'

Spark is one of the hottest technologies in big data analysis right now, and with good reason. If you work for, or you hope to work for, a company that has massive amounts of data to analyze, Spark offers a very fast and very easy way to analyze that data across an entire cluster of computers and spread that processing out. This is a very valuable skill to have right now.

My approach in this book is to start with some simple examples and work our way up to more complex ones. We'll have some fun along the way too. We will use movie ratings data and play around with similar movies and movie recommendations. I also found a social network of superheroes, if you can believe it; we can use this data to do things such as figure out who's the most popular superhero in the fictional superhero universe. Have you heard of the Kevin Bacon number, where everyone in Hollywood is supposedly connected to a Kevin Bacon to a certain extent? We can do the same thing with our superhero data and figure out the degrees of separation between any two superheroes in their fictional universe too. So, we'll have some fun along the way and use some real examples here and turn them into Spark problems. Using Apache Spark is easier than you might think and, with all the exercises and activities in this book, you'll get plenty of practice as we go along. I'll guide you through every line of code and every concept you need along the way. So let's get started and learn Apache Spark.