LazyProgrammer - Big Data, MapReduce, Hadoop, and Spark with Python

Once you master the MapReduce paradigm, coding with MapReduce will simply just become regular programming. Because MapReduce is designed to scale to big data, there is a lot of overhead involved in setting things up. Once you understand this overhead (both in terms of computational overhead and effort on your part), big data programming is just programming.

You may have read that MapReduce proceeds in 2 steps: mapping, followed by reducing, which are done by processes called mappers and reducers, respectively. I dont find these descriptions particularly helpful. They certainly dont get you any closer to actually coding MapReduce jobs. They are more like a high-high-high-high-high level overview that perhaps a CEO who is not interested in the technical details would need to know. As a programmer, the above description is not useful. It is best to think about what the data looks like as we proceed from one step to the next.

The most straightforward way to think about big data processing is to think about text files. Text files will consist of a number of lines, each of which can be read in one at a time. In Python, it might look something like this: for line in open(input.txt): do_something(line) As you can see, this time complexity of this code is O(N). It will take twice as long to process a file that is twice as large. An obvious solution to this would be to break up the work. If we have a dual-core CPU, we could split up the file into 2 files of equal size: input1.txt and input2.txt.

Run the same program twice on each of the files, each will run on a separate core, thus taking half the time. If we have a quad-core CPU, we could split up the file into 4 parts and it would take a quarter of the time. Before MapReduce came about, this is how you would scale. Just split up the data, and spread out the computation across different CPUs, even different machines (if they have network capabilities). The problem with this is its not systematic enough. Its not general enough that we can have just one framework that takes care of all the details for us.

Some of these details might include: - One machine dies during computation. How do you know what it was working on so that you can then pass that job to another machine? - How would you even keep track of the fact that it died? - One machine is taking significantly longer than the others. How can you make sure youre spreading the work evenly? The MapReduce framework includes a master node that keeps track of all these things, and even runs an HTTP server so that you can track the progress of a job. To understand the MapReduce framework as an algorithm, you dont need to understand which machine does what. What you do need to understand is the map function and the reduce function. [Side note: We focus on implementation in this book, because when you join a new company as a big data hire, these systems are likely already setup by the DevOps team. [Side note: We focus on implementation in this book, because when you join a new company as a big data hire, these systems are likely already setup by the DevOps team.

Your job will be the coding part.] The best way is to learn by example, so that is what we are going to do now. The canonical example for MapReduce is word counting. You have a huge number of text documents (notice that this implies the data is already split up, not just one huge file), and you want to output a table showing each word and its corresponding count. If you were to do this in regular Python, it would look something like this: # count the words word_counts = {} for f in files: for line in open(f): words = line.split() for w in words: word_counts[w] = word_counts.get(w, 0) + 1 # display the result for w, c in word_counts.iteritems(): print %s\t%s % (w, c) Note that in big data processing, we often print data out in a table format. Implementation-wise, this means we use CSV or some other character-delimited format (tabs and pipes are also common). What would this look like in MapReduce? mapper.py for line in input: words = line.split() for w in words: print %s\t%s % (w, 1) reducer.py for key, values in input: print %s\t%s % (key, sum(values)) input here is abstracted and can be different in different contexts.