Edd Wilder-James - Planning for Big Data

Beijing Cambridge Farnham Kln Sebastopol Tokyo

In February 2011, over 1,300 people came together for the inaugural OReilly Strata Conference in Santa Clara, California. Though representing diverse fields, from insurance to media and high-tech to healthcare, attendees buzzed with a new-found common identity: they were data scientists. Entrepreneurial and resourceful, combining programming skills with math, data scientists have emerged as a new profession leading the march towards data-driven business.

This new profession rides on the wave of big data. Our businesses are creating ever more data, and as consumers we are sources of massive streams of information, thanks to social networks and smartphones. In this raw material lies much of value: insight about businesses and markets, and the scope to create new kinds of hyper-personalized products and services.

Five years ago, only big business could afford to profit from big data: Walmart and Google, specialized financial traders. Today, thanks to an open source project called Hadoop, commodity Linux hardware and cloud computing, this power is in reach for everyone. A data revolution is sweeping business, government and science, with consequences as far reaching and long lasting as the web itself.

Every revolution has to start somewhere, and the question for many is how can data science and big data help my organization? After years of data processing choices being straightforward, theres now a diverse landscape to negotiate. Whats more, to become data-driven, you must grapple with changes that are cultural as well as technological.

The aim of this book is to help you understand what big data is, why it matters, and where to get started. If youre already working with big data, hand this book to your colleagues or executives to help them better appreciate the issues and possibilities.

I am grateful to my fellow OReilly Radar authors for contributing articles in addition to myself: Alistair Croll, Julie Steele and Mike Loukides.

Edd Dumbill

Program Chair, OReilly Strata Conference

February 2012

Chapter 1. The Feedback Economy

By Alistair Croll

Military strategist John Boyd spent a lot of time understanding how to win battles. Building on his experience as a fighter pilot, he broke down the process of observing and reacting into something called an Observe, Orient, Decide, and Act (OODA) loop. Combat, he realized, consisted of observing your circumstances, orienting yourself to your enemys way of thinking and your environment, deciding on a course of action, and then acting on it.

The Observe, Orient, Decide, and Act (OODA) loop. Larger version available here..

The most important part of this loop isnt included in the OODA acronym, however. Its the fact that its a loop . The results of earlier actions feed back into later, hopefully wiser, ones. Over time, the fighter gets inside their opponents loop, outsmarting and outmaneuvering them. The system learns.

Boyds genius was to realize that winning requires two things: being able to collect and analyze information better, and being able to act on that information faster, incorporating whats learned into the next iteration. Today, what Boyd learned in a cockpit applies to nearly everything we do.

Data-Obese, Digital-Fast

In our always-on lives were flooded with cheap, abundant information. We need to capture and analyze it well, separating digital wheat from digital chaff, identifying meaningful undercurrents while ignoring meaningless social flotsam. Clay Johnson argues that we need to go on an information diet, and makes a good case for conscious consumption. In an era of information obesity, we need to eat better. Theres a reason they call it a feed, after all.

Its not just an overabundance of data that makes Boyds insights vital. In the last 20 years, much of human interaction has shifted from atoms to bits. When interactions become digital, they become instantaneous, interactive, and easily copied. Its as easy to tell the world as to tell a friend, and a days shopping is reduced to a few clicks.

The move from atoms to bits reduces the coefficient of friction of entire industries to zero. Teenagers shun e-mail as too slow, opting for instant messages. The digitization of our world means that trips around the OODA loop happen faster than ever, and continue to accelerate.

Were drowning in data. Bits are faster than atoms. Our jungle-surplus wetware cant keep up. At least, not without Boyds help. In a society where every person, tethered to their smartphone, is both a sensor and an end node, we need better ways to observe and orient, whether were at home or at work, solving the worlds problems or planning a play date. And we need to be constantly deciding, acting, and experimenting, feeding what we learn back into future behavior.

Were entering a feedback economy.

The Big Data Supply Chain

Consider how a company collects, analyzes, and acts on data.

The big data supply chain. Larger version available here..

Lets look at these components in order.

Data collection

The first step in a data supply chain is to get the data in the first place.

Information comes in from a variety of sources, both public and private. Were a promiscuous society online, and with the advent of low-cost data marketplaces, its possible to get nearly any nugget of data relatively affordably. From social network sentiment, to weather reports, to economic indicators, public information is grist for the big data mill. Alongside this, we have organization-specific data such as retail traffic, call center volumes, product recalls, or customer loyalty indicators.

The legality of collection is perhaps more restrictive than getting the data in the first place. Some data is heavily regulated HIPAA governs healthcare, while PCI restricts financial transactions. In other cases, the act of combining data may be illegal because it generates personally identifiable information (PII). For example, courts have ruled differently on whether IP addresses arent PII, and the California Supreme Court ruled that zip codes are. Navigating these regulations imposes some serious constraints on what can be collected and how it can be combined.

The era of ubiquitous computing means that everyone is a potential source of data, too. A modern smartphone can sense light, sound, motion, location, nearby networks and devices, and more, making it a perfect data collector. As consumers opt into loyalty programs and install applications, they become sensors that can feed the data supply chain.

In big data, the collection is often challenging because of the sheer volume of information, or the speed with which it arrives, both of which demand new approaches and architectures.

Ingesting and cleaning

Once the data is collected, it must be ingested. In traditional business intelligence (BI) parlance, this is known as Extract, Transform, and Load (ETL): the act of putting the right information into the correct tables of a database schema and manipulating certain fields to make them easier to work with.