Felix Liao - Smart Data Discovery Using SAS Viya

Chapter 1: Why Smart Data Discovery?

Introduction

As information workers, our ability to leverage data and extract insights in order to make critical business decisions is fundamental to our success as individuals and the organizations that we work for. Regardless of whether you are an executive, departmental decision maker, or an analyst, the need to leverage data and analytical techniques effectively in order make business decisions is now pervasive throughout every part of an organization.

From the organizations perspective, the historical approach of relying solely on statisticians or decision support specialists to prepare and analyze data is no longer a workable approach. Organizations today must involve everyone in their analytics efforts especially those closest to core business functions to truly leverage data for maximum strategic and tactical advantage.

The good news for both us as individuals and the organizations that we work for is that tremendous advancements in computer hardware and software in recent years have allowed us to collect more data than ever before. Furthermore, with the addition of advanced analytics and machine learning capabilities, modern analytics tools are now easier to use and have never been more powerful. These shifts have made data more accessible and true self-service analytics a reality today.

One area of analytics that has made a significant impact in recent years is self-service data visualization. These easy-to-use data visualization and exploration tools enable any information workers today to assemble data rapidly, explore hypotheses visually, and find new insights quickly. Data visualization tools empower business users and accelerate the process of insight discovery by reducing the need for statisticians, data modelers, or IT specialists. By shifting the process of insight discovery closer to the business and subject matter experts, it has enabled more timely and relevant insights to be discovered and acted upon.

The greatest value of a picture is when it forces us to notice what we never expected to see.

John Tukey

Not only have these new data visualization tools accelerated the process of insight discovery, they have also allowed business users to ask more complex and forward-looking questions. These powerful, visual-based data discovery tools have revolutionized the traditional business intelligence solution space and led the way in terms of self-service analytics. Never before has it been easier for individual users to explore and visualize data for powerful insight with such ease.

With growing awareness and understanding of advanced data visualizations techniques, business users are now increasingly asking more complex questions, conduct more forward-looking analysis and eager to move beyond basic charts and graphs for answers. Enter the era of smart data discovery and the rise of citizen data scientist. Smart data discovery extends beyond the realm of traditional charts and visualization techniques with embedded machine learning techniques and algorithms. This new, augmented approach to data discovery leverages new visualization frameworks and automated machine learning capabilities to empower a new generation of users often described as citizen data scientists. Smart data discovery enables deeper insight discovery and empowers these new citizen data scientists to conduct deeper investigation, ask forward-looking questions, and develop valuable predictive insights.

While smart data discovery holds tremendous promise, a new approach is needed in order to fully release its potential. At the core of this new approach is the recognition that citizen data scientists will need to be equipped with a new set of knowledge and skills, including the following:

1. Business Domain Knowledge Smart data discovery needs to be built on a deeper understanding of the relevant business context and problem domains. Smarter insights can only come from asking more relevant and intelligent questions.

2. Analytics Techniques A high level of familiarity with various analytical techniques and principles is required. While a PhD in Statistics is not necessary, a sound understanding of fundamental statistics and machine learning principles and techniques will be needed.

3. Product Know-How Finally, it is about having access to the right tools and the necessary skills to bring it all together in a timely manner.

As depicted in Figure 1.1 below, these three skill sets in isolation are valuable, but when combined to solve a specific business problem, a new level of analytics and insight can be achieved in this case, the sum is greater than its parts.

Figure 1.1: Smart Data Discovery Requirements

This book will help you navigate these intersecting knowledge domains and empower you to ask more complex questions by illustrating the key components of a smart data discovery process. We will highlight fundamental statistical concepts and how to leverage the relevant features. Most importantly, we will also be using real examples and applications to bring these concepts together.

SAS Viya, the latest evolution of the SAS platform will be used to demonstrate these examples throughout this book. An introduction to relevant features and functionalities that are needed in a smart data discovery process will be provided, followed by an explanation on how to leverage and interpret the various charts and outputs from SAS Viya.

Smart data discovery has the potential to shift an organizations overall analytic maturity, accelerate its analytical efforts, and create a much bigger analytics workforce. From an individual perspective, it has the potential to transform the way you view data, conduct data discovery processes, and think about how complex business problems can be solved. In many ways, we are just at the start of this revolution, and I am hopeful that this book will help you and your organization lead the way in terms of realizing the true potential of data and analytics.

Why Smart Data Discovery Now?

Traditional Business Intelligence (BI) and data visualization tools do a great job of slicing and dicing data in order to help answer questions such as what happened and what is happening. These tools can also provide valuable dashboards and reports for the purpose of insights sharing and communication. However, they cannot easily identify correlating factors or help predict future outcomes. These outcomes might include: which customers will respond to a promotion? Who will churn to a competitor? And when will a piece of equipment fail? In a modern environment where businesses are no longer content with simply analyzing data from the past, but instead would also like to gain insights into the future, a new approach is clearly needed.