McCormick - IBM SPSS Modeler cookbook

Our company, ISL was a provider of Artificial Intelligence tools and technology to organizations developing advanced software solutions. By 1992, what had started as a casual interest from our clients in applying some of our toolsthe machine learning modulesto their historic data had evolved into a promising practice in what was to become known as data mining. This was developing into a nice line of business for us, but was frustrating in a couple of ways:

First, we'd always intended that ISL should be a software supplier. Yet here we were, because of the complexity of the technologies involved, providing data mining on a consulting services basis.

Secondly, we were finding that data mining projects involved a lot of hard work, and that most of that work was boring. Unearthing significant patterns and delivering accurate predictionsthat part was fun. But most of our effort went on mundane tasks such as manipulating data into the formats required by the various modules and algorithms we applied.

So we built Clementineto make our job easier and allow us to focus on the interesting parts of projects, and to give us a tool we could provide to our clients. When the first prototypes were ready, we tested them by using them to re-run projects we'd previously executed manually. We found work that had previously taken several weeks was now reduced to under an hour; we'd obviously got something right.

As the embryonic data mining market grew, so did our business. We saw other vendors, with deeper pockets and vastly more resources than little ISL, introduce data mining tools, some of which tried to emulate the visual style of the Clementine's user interface. We were relieved when, as the inevitable shoot-outs took place, we found time and time again evaluators reporting that our product had a clear edge, both in terms of productivity and the problem-solving power it gave to analysts.

On reflection, the main reasons for our success were that we got a number of crucial things right:

Clementine's design and implementation, from the ground up, was object-oriented. Our visual programming model was consistent and "pure"; learn the basics, and everything is done in the same way.

We stuck to a guiding principle of, wherever possible, insulating the user from technology details. This didn't mean we made it for dummies; rather, we ensured that default configurations were as sensible as possible (and in places, truly smartwe weren't AI specialists for nothing), and that expert options such as advanced parameter settings were accessible without having to drop below the visual programming level.

We made an important design decision that predictive models should have the same status within the visual workflow as other tools, and that their outputs should be treated as first-order data. This sounds like a simple point, but the repercussions are enormous. Want more than the basic analysis of your model's performance? No problemrun its output through any of the tools in the workbench. Curious to know what might be going on inside your neural network? Use rule induction to tell you how combinations of inputs map onto output values. Want to have multiple models vote? Easy. Want to combine them in more complex ways? Just feed their inputs, along with any data you like, into a supermodel that can decide how best to combine their predictions.

The first two give productivity, plus the ability to raise your eyes from the technical details, think about the process of analysis at a higher level, and stay focused on each project's business objectives. Add the third, and you can experiment with novel and creative approaches that previously just weren't feasible to attempt.

So, 20 years on, what do I feel about Clementine/Modeler? A certain pride, of course, that the product our small team built remains a market leader. But mainly, over the years, awe at what I've seen people achieve with it: not just organizations who have made millions (sometimes, even billions) in returns from their data mining projects, but those who've done things that genuinely make the world a better place; from hospitals and medical researchers discovering new ways to diagnose and treat pediatric cancer, to police forces dynamically anticipating levels of crime risk around their cities and deploying their forces accordingly, with the deterrent effect reducing rates of murder and violent crime by tens of percent. And also, a humble appreciation for what I've learned over the years from users who took what we'd createda workbench and set of toolsand developed, refined, and applied powerful approaches and techniques we'd never thought of.

The authors of this book are among the very best of these exponents, gurus who, in their brilliant and imaginative use of the tool, have pushed back the boundaries of applied analytics. By reading this book, you are learning from practitioners who have helped define the state of the art.

When Keith McCormick approached me about writing this foreword, he suggested I might like to take a "then" and "now" perspective. This is certainly an interesting "now" in our industry. The advent of Big Datahuge volumes of data, of many varieties and varying veracity, available to support decision making at high velocitypresents unprecedented opportunities for organizations to use predictive analytics to gain value. There is a danger, though, that some of the hype around this will confuse potential adopters and confound their efforts to derive value for their business. One common misconception is that you just get all data you can together, and then poke around in the hope of finding something valuable. This approachtell me something interesting in this datawas what we always considered "the data mining question from hell", and is very unlikely to result in real, quantifiable benefit. Data mining is first and foremost a business activity, and needs to be focused on clear business objectives and goals, hence the crucial business understanding phase in CRISP-DM that starts every data mining project.