Dejan Sarka [Dejan Sarka] - Data Science with SQL Server Quick Start Guide

A long-term data science project is somehow never finished. It has its own complete life cycle. This virtuous cycle includes the following steps:

1. Identify the business problem
2. Use data mining and machine learning techniques to Transform the data into actionable information
3. Act on the information
4. Measure the result

Data science is not a product. Data science gives you a platform for continuous learning on how to improve your business. In order to learn how to exploit data mining maximally, you need to measure the results of your actions based on the information extracted with data mining. Measurement provides the feedback for continuously improving results. You can see the life cycle in the following diagram:

Let me give you an example. For credit card issuers and online banking, fraud detection is quite a common task. You want to identify fraudulent transactions as quickly as possible to minimize the damage. You realize that you get some frauds in your system. You identified a problem, so this is the Identify part. Then, you use your existing data and implement a data science algorithm on it to get a model. This is the Transform part. Then, you use the model to perform online predictions and divert possible fraudulent transactions for an inspection immediately when they appear in your system. This is the Act part. Now, you need to close the cycle. You realize that fraudsters learn, and the patterns of frauds can change over time. Therefore, the model you deployed in production might become obsolete. You need to measure the quality of the predictions over time: perform the Measure part of the life cycle. When the quality drops, it is time to refine your model. This means you just identified a problem.

Of course, not every project is so complex as the fraud detection is. For a simple one-time marketing campaign, you do not close the loop with the measuring part; once you use the model in production, the project can be closed.

The second step in this life cycle, the step where you transform your data to an actionable knowledge, is the part when you do the most complex work. This is the part that this book focuses on. This part is further divided into steps that can be execute in loops. The Cross Industry Standard Process for Data Mining (CRISP) model is an informally standardized process for the Transform part. It splits the process in six phases. The sequence of the phases is not strict. Moving back and forth between different phases is always required. The outcome of each phase determines the next phase (or particular task of a phase) that has to be performed. The arrows indicate the most important and frequent dependencies between phases. The following diagram depicts the CRISP model:

The six CRISP phases should finish with some deliverables. The phases with typical deliverables include the following:

• Business understanding: Data-mining problem definition
• Data understanding: Data quality reports, descriptive statistics, graphical presentations of data, and so on
• Data preparation: Cleansed training and evaluation datasets, including derived variables
• Modeling: Different models using different algorithms with different parameters
• Evaluation: Decision whether to use a model and which model to use
• Deployment: End-user reports, OLAP cube structure, OLTP soft constraints, and so on

You can learn more about this model at https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining.