YASSINE MOUSAIF - The Data Engineering Cookbook: Mastering The Plumbing Of Data Science

What do you actually need to learn to become an awesome data engineer?Look nofurther, youll find it here.
If you are looking for AI algorithms and such data scientist things, this book is not foryou.

How to use this document:
First of all, this is not a training!This cookbook is a collection of skills that I valuehighly in my daily work as a data engineer. Its intended to be a starting point for youto find the topics to look into and become an awesome data engineer.

You are going to find Five Types of Content in this book: Articles I wrote, links tomy podcast episodes (video & audio), more than 200 links to helpful websites I like, dataengineering interview questions and case studies.

This book is a work in progress!
As you can see, this book is not finished.Im constantly adding new stuff and doingvideos for the topics. But obviously, because I do this as a hobby my time is limited.You can help making this book even better.

Help make this book awesome!
If you have some cool links or topics for the cookbook, please become a contributor onGitHub: https://github.com/andkret/Cookbook. Pull the repo, add them and create apull request. Or join the discussion by opening Issues. You can also write me an emailany time to plumbersofdatascience@gmail.com. Tell me your thoughts, what you value,what you think should be included, or correct me where I am wrong.

This Cookbook is and will always be free!
I dont want to sell you this book, but please support what you like and join my Patreon:https://www.patreon.com/plumbersofds

Check out this podcast episode where I talk in detail why I decided to share all thisinformation for free: #079 Trying to stay true to myself and making the cookbook publicon GitHub

Podcast Episode: #050 Data Engineer, Scientist or Analyst - Which One Is For You?In this podcast we talk about the differences between data scientists, analysts andengineers. Which are the three main data science jobs. All three are super important. This makes it easy to decide
YouTubeClick here to watch
AudioClick here to listen

Table 2.1: Podcast: 050 Data Engineer, Scientist or Analyst - Which One Is For You? Data scientists arent like every other scientist.
Data scientists do not wear white coats or work in high tech labs full of science fictionmovie equipment. They work in offices just like you and me.
What differs them from most of us is that they are math experts. They use linear algebraand multivariable calculus to create new insight from existing data.
How exactly does this insight look?
Heres an example:
An industrial company produces a lot of products that need to be tested before shipping.
Usually such tests take a lot of time because there are hundreds of things to be tested.All to make sure that your product is not broken.
Wouldnt it be great to know early if a test fails ten steps down the line? If you knewthat you could skip the other tests and just trash the product or repair it.
Thats exactly where a data scientist can help you, big-time. This field is called predictiveanalytics and the technique of choice is machine learning.
Machine what? Learning?Yes, machine learning, it works like this:

You feed an algorithm with measurement data.It generates a model and optimises itbased on the data you fed it with. That model basically represents a pattern of how yourdata is looking. You show that model new data and the model will tell you if the datastill represents the data you have trained it with. This technique can also be used forpredicting machine failure in advance with machine learning. Of course the whole processis not that simple.

The actual process of training and applying a model is not that hard.A lot of workfor the data scientist is to figure out how to pre-process the data that gets fed to thealgorithms.

In order to train a algorithm you need useful data. If you use any data for the trainingthe produced model will be very unreliable.

A unreliable model for predicting machine failure would tell you that your machine isdamaged even if it is not. Or even worse: It would tell you the machine is ok even whenthere is an malfunction.

Model outputs are very abstract.You also need to post-process the model outputs toreceive health values from 0 to 100.
Figure 2.1: The Machine Learning Pipeline Data Engineers are the link between the managements big data strategy and the datascientists that need to work with data.
What they do is building the platforms that enable data scientists to do their magic.
These platforms are usually used in five different ways:
Data ingestion and storage of large amounts of data
Algorithm creation by data scientists
Automation of the data scientists machine learning models and algorithms forproduction use
Data visualisation for employees and customers

Most of the time these guys start as traditional solution architects for systemsthat involve SQL databases, web servers, SAP installations and other standardsystems.

But to create big data platforms the engineer needs to be an expert in specifying, setting up and maintaining big data technologies like: Hadoop, Spark, HBase, Cassandra,MongoDB, Kafka, Redis and more.

What they also need is experience on how to deploy systems on cloud infrastructure likeat Amazon or Google or on-premise hardware.

Podcast Episode: #048 From Wannabe Data Scientist To Engineer My JourneyIn this episode Kate Strachnyi interviews me for her humans of data science podcast.We talk about how I found out that I am more into the engineering part of datascience.
YouTubeClick here to watch
AudioClick here to listen

Table 2.2: Podcast: 048 From Wannabe Data Scientist To Engineer My Journey

For a good company it is absolutely important to get well trained data engineers and datascientists. Think of the data scientist as the professional race car driver. A fit athletewith talent and driving skills like you have never seen.

What he needs to win races is someone who will provide him the perfect race car to drive.Thats what the solution architect is for.
Like the driver and his team the data scientist and the data engineer need to work closelytogether. They need to know the different big data tools inside and out.
Thats why companies are looking for people with Spark experience.It is a commonground between both that drives innovation.

Spark gives data scientists the tools to do analytics and helps engineers to bring the datascientists algorithms into production.After all, those two decide how good the dataplatform is, how good the analytics insight is and how fast the whole system gets into aproduction ready state.

Why this is important: Without coding you cannot do much in data engineering. I cannotcount the number of times I needed a quick Java hack.
The possibilities are endless:
Writing or quickly getting some data out of a SQL DB
Testing to produce messages to a Kafka topic
Understanding the source code of a Java Webservice