Rick van der Lans - Data Virtualization for Business Intelligence Systems: Revolutionizing Data Integration for Data Warehouses

The classic data warehouse and business intelligence architecture relies on a repository of quality, integrated data at its core. In the very early days of business intelligence, we struggled with manual processes to extract data from multiple operational systems, combine the data, fix any errors, fill in missing fields, remove duplicate data, and finally load the integrated data into a database, creating a physical data warehouse, or single source of data, for reporting and analytics.

Shortly thereafter came the technological innovation of extraction, transformation, and load (ETL) tools, which automated many manual data integration tasks in a reliable and repeatable fashion. ETL tools greatly improved the overall process of creating a data warehouse. They included data quality technology to further improve the value of the integrated data for decision making. To this day, ETL tools remain a major mechanism for creating physical stores of historical data for business intelligence.

Recently, two significant trends are causing business intelligence architects to rethink their ETL and data management infrastructures: operational business intelligence and the advent of big data analytics. Lets look at operational business intelligence first. Most business intelligence environments start out producing historical reports and analytics about what has happened. Historical data can also be used for predicting what will happen, but it does not fully support real-time decisioning or operational business intelligence.

As enterprises started demanding the ability to make intra-day decisions based on current or low-latency data, we sped up the overall ETL process via change data capture, trickle feeds, and minibatches of operational data. These approaches reduced the latency of the data in our data warehouses from days and hours to minutes, but they were still not fast enough for true real-time decision making. Business intelligence implementers came to realize that classic ETL processing had reached its limits, and a new form of data integration was in order.

It is a similar story for big data and its associated analytics. Examples of big data include social and text analytics, sensor data, and event or in-motion data. Much of big data is unstructured or, more accurately, occurs in multiple formats. It does not have the traditional and predictable structures found in typical operational systems. Its also massive in volume, relative to previous standards. For many data warehouse implementers, big data poses significant integration challenges.

Truth be told, much big data may not need to reside in a formal data warehouse. Often, it is used for experimental or investigative analytics. Even so, there may be a need to combine some of this data with the data warehouse data. How can we effectively accommodate the demand for operational business intelligence and big data and extend business intelligence architectures without disrupting existing ETL processes? The answer is data virtualization.

I have known Rick van der Lans for many years. His articles and research papers often educate me because they always give me interesting alternatives and innovative twists to traditional thinking. Rick forces me to reevaluate all I held as true. And this is certainly true of his latest effort: Data Virtualization for Business Intelligence Systems: Revolutionizing Data Integration for Data Warehouses.

Data virtualization has become a must-have technology for todays business intelligence implementers. As with any new technology, there are many questions about how to implement it, when to use it, and what challenges and pitfalls to avoid. Rick covers these issues and more in this detailed and practical how-to guide. I will be referring to it for years to come. I know you will as well.

Claudia Imhoff

President of Intelligent Solutions, Inc.,

Founder of the Boulder BI Brain Trust (BBBT)