Lewis Gavin - What Is Data Engineering?

• 2019-12-04: First Release

Data engineering is becoming more and more popular but is actually a role that has existed for quite some time. Data engineers have been branded in a whole host of different ways in the past, including as database administrators (DBAs), data analysts, business intelligence engineers, database developers, and more.

Recently, although the core role itself has remained fairly constant, the term data engineering has taken on slightly different connotations due to several changes in the market. But at its heart, data engineering is all about the movement, manipulation, and management of data.

This trend has snowballed from the large businesses mentioned in the first paragraph to many others as they come to understand the importance and power of data and not in the traditional sense of just storing basic application and customer data, but storing every event in as little detail as possible. The more data you have, the more knowledge you can redeem from it. You can see in that the global datasphere grew to 40 zettabytes (ZB) in 2019 (1 ZB is around a billion terabytes) and will continue to grow as more businesses harvest its power. This growth in data production has led to major advancements in data technology and data jobs, especially in the fields of data engineering and data science.

Figure 1. Global data size by year

The existing technologies and previously used infrastructure werent suited to store this new wave of event data, as the volume became very large, very quickly. This meant that not only would systems eventually slow down but the existing payment models surrounding them meant businesses would have to pay very large amounts of money to support them.

The emergence of big data technologies is where data engineering picked up its modern definition. The term big data came into use as early as 1998. John Mashey used it to define the growth in data storage and data production. In 2001 Doug Laney produced a report for Gartner that took this definition one step further and wrapped it up into the three vees: volume, velocity, and variety.

To deal with the three vees, data warehouses became data lakes and technologies such as Hadoop were developed to help store and process this data on commodity hardware. It seemed like a no brainer for businesses to adopt big data solutions due to the promise of easily scalable storage capacity and faster processing at a much smaller price.

This gave birth to the modern data engineer, a hybrid of a data engineer and a software engineer. Eventually, the marketplace caught up, and Facebook created a technology called Hive for Hadoop that allows SQL to be used. Businesses could then migrate existing skill sets over. Businesses like Facebook and LinkedIn continued to push the boundaries to make the development of big data solutions easier.

However, the differences between the traditional data engineer and a software engineer have become ever smaller as the integration of data capture and data processing has become embedded into core parts of services, instead of just the business information (BI) that sits around them. Finding efficient ways to capture data in real time, process it, generate user-bespoke content, and store it has lead to huge growth in the number of engineers that understand both software and data, and it is now the rule rather than the exception that data engineers have these skill sets.

To leverage all this new data, businesses also needed new methods of processing data. Machine learning and artificial intelligence (AI) became popular very quickly due to the availability of data sets that are large enough to build useful models. Data science is the emerging field for these techniques and is often seen as the money maker over data engineering; however, when the two roles coexist, they complement each other.

Data scientists require data from many sources that need manipulation and aggregation before being used. This is where the marriage with data engineering comes into play, as there is a large amount of engineering required to build an accurate machine learning model. Add on the difficulty of dealing with extra large data sets and these data manipulation tasks quickly become tedious for data scientists, who naturally prefer to (and should) spend their time on increasing the accuracy of their models.

This marriage with data science might explain why data engineering is currently one of the most , Google Trends also shows that interest in the search term data engineer is on the rise and is currently at its all-time highest.