Richard Nuckolls - Azure Data Engineering: Real-time, streaming, and batch analytics MEAP V08

• What is data engineering?

• What do data engineers do?

• How does Microsoft define data engineering?

• What tools does Azure provide for data engineering?

Collecting the data seems like a simple activity. Take reporting web site traffic. A single user, during a site in a web browser, requests a page. A simple site could respond with an HTML file, a CSS file, and an image. This example could represent one event, three events, or four events. What if there is a page redirect? That is another event. What if we want to log the time taken to query a database? What if we retrieve some items from cache? All of these pieces of data are commonly logged data points today.

Now add more user interaction, like a comparison page with multiple sliders. Each move of the slide logs a value. Tracking user mouse movement returns hundreds of coordinates. Consider a connected sensor with a 100Hz sample rate. It can easily record over 8 million measurements a day. When you start to scale to thousands and tens of thousands of simultaneous events every point in the pipeline must be optimized for speed until the data comes to rest. Once at rest the data must remain secure.

Data engineering is the practice of building data storage and processing systems. Robert Chang, in his "A Beginners Guide to Data Engineering," describes the work as designing, building, and maintaining data warehouses. Data engineering creates scalable systems which allow analysts and data scientists to extract meaningful information from the data.

Most businesses have multiple sources generating data. Manufacturing companies track the output of the machines, the output of employees, the output of their shipping departments. Software companies track their user actions, their software bugs per release, their developer output per day. Service companies check number of sales calls, time to complete tasks, usage of parts stores, and cost per lead. Some of this is small scale; some of it is large scale.

Analysts and managers might operate on narrow data sets, but large enterprises increasing want to find efficiencies across divisions, or find root causes in multi-faceted systems failures. In order to extract value from these disparate sources of data, engineers have built large scale storage systems as a single repository of data. A software company may implement centralized error logging. The service company may integrate their CRM, billing, and finance systems. Engineers need to plan for the ingestion pipeline, the storage Systems, and reporting services across multiple groups of stakeholders.

As a first step data consolidation means a large relational database. Analysts review reports, CSV files, and even spreadsheets in Excel, in an attempt to get clean and consistent data. Often developers or database administrators prepare scripts to import the data into databases. In the best case experienced database administrators define common schema and plan partioning and indexing. The database enters production. Data collection commences in earnest.