Eagle Nathan - Reality mining : using big data to engineer a better world

Reality Mining

Reality Mining

Using Big Data to Engineer a Better World

Nathan Eagle and Kate Greene

The MIT Press
Cambridge, Massachusetts
London, England

2014 Massachusetts Institute of Technology

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

Library of Congress Cataloging-in-Publication Data

Eagle, Nathan

Reality mining: using big data to engineer a better world/ Nathan Eagle and Kate Greene

p. cm.

Includes bibliographical references and index.

ISBN 978-0-262-02768-7 (hardcover : alk. paper)

ISBN 978-0-262-32457-1 (retail e-book)

1. Data mining. 2. Big data. 3. Computer networksSocial aspects. 4. Information scienceSocial aspects. 5. Information scienceStatistical methods. I. Greene, Kate, 1979 II. Title.

QA76.9.D343E24 2014

006.312dc23

2013047165

Introduction

Big Data is all the rage. Conferences, books, research papers, and entrepreneurial interest on the topic abound. And with good reason: the idea of mining meaning from a previously unfathomable amount of data to identify clear trends and even to predict the future is certainly enchanting. But as all the conferences, books, research papers, and business plans illustrate, figuring out how to grapple with data on this scale and make good use of them is no simple task.

As we define it, Big Data is the collected bits of information produced from interactions that people and objects have with the digital, often networked world. The data can represent a single variable from an individual collected over years or multiple variables taken from hundreds of millions of people collected in an instant. Big Data can be long in time, voluminous in subject number (n), or broad in scope. Or it can be some combination of these three characteristics.

Thanks to the confluence of a variety of technological factors, Big Data is a fact of our modern world. Mobile, networked computers with powerful processors sit in your pocket, collecting data, crunching numbers, and sending bits to remote servers. Cloud computing and the ever-increasing density of digital storage provides a home for all the information. Moreover, stream-processing paradigms allow data to be processed over distributed machines. Availability of programming models for large data sets, including MapReduce and the open-source Hadoop, have made it possible to make sense of the torrents of incoming information.

Big Data has been called the digital exhaust or the digital footprints we leave in the wake of everyday activity; it is the metadata of our lives. To some, the phrase evokes fear of a world without privacy, of companies knowing more about us than we know about ourselves, of governments keeping tabs on people who the governments believe could pose a threat to their power. To others, Big Data is a pot of gold at the end of a rainbow of databases, an opportunity to capitalize on the next big information technology trend. And still others believe that some good can actually be gleaned from the exabytes of data produced daily by people all over the world.

We, the authors of this book, are technologists who fall into the latter category. We believe that if considered in a responsible, respectful, and context-aware way, Big Data can aid public health, guide better individual decision making, foster the sharing of useful knowledge, and increase the rate of innovation. The era of Big Data is here, and it isnt going to be over anytime soon. It is important to ensure that individual liberties and freedoms are maintained, that privacy isnt ignored, and that consumers are made aware of who accesses their data, when, and for which purposes. We believe that within a conscientious context of data collection, its possible to use Big Data to engineer better systems and potentially a better world. We use an approach we call Reality Mining, which is not only about analyzing Big Data, but about ensuring that the analysis reflects the reality of the situation and the people involved, while being consistent with a conscientious data-collection approach.

The goal of this book, then, is to explore the positive potential of Big Dataspecifically, to show how Reality Mining can be used to engineer better social systems. This means we introduce ideas meant to transcend simple, descriptive analytics such as bar graphs of pedometer data. We discuss ways to turn the data from observational maps of crimes or of disease outbreaks into meaningful actions or policies. We consider systems designed to use global, anonymous data. And we ask questions such as, How can an early-warning system for disease epidemics be deployed in developing countries with limited public-health resources? In essence, we want to explore how the use of Big Data can change peoples lives for the better.

This book is divided into five parts. Each part addresses challenges and opportunities inherent to Big Data on various scales of data collection. Much in the same way that Powers of Ten, the 1977 short film by Charles Eames and Ray Eames, looks at the very small and very large parts of the universe, we address Big Data at small to increasingly larger scales.

We start our Reality-Mining journey at the individual scale, with data generated by and applied to a single person: the individual. From there, we move up a level to the neighborhood and organization, then to the city, followed by the nation, and finally the world. Admittedly, this framework is not absolute, nor is it applicable to all scenarios. Big Data collected at a particular scale can be applied at various other scales, of course. When appropriate, we mention some examples of this varied application. But as a way of thinking about the challenges inherent to collecting Big Data and making use of it, our five-level structure is a manageable one.

Each of the five parts in this book is divided into two chapters. The first chapter in each part details the type of data collected, the manner in which they are collected, and how, if possible, a reader might gain access to the data. The second chapter in each part details the applications and systems that have been built or we believe could be built based on the data.

Thus, the first chapter in each part serves as a guide to thinking about the various types of mineable data and how one might go about acquiring access to these data. Approaches to data acquisition range from creating a mobile app that collects sleep data to subscribing to a provider of air-traffic data to using the free and simple Google-based tools to parse search data. And in some cases, when data arent easily accessible to most people, such as call data records from mobile phones, we offer suggestions for gaining limited access to the data or for finding emerging alternatives to the actual data.

Once you understand the type of data that can be mined, we discuss Reality-Mining applications that make use of it. Some of these systems exist, some are in early stages, and others havent been explored at all. We provide information on just a handful of available applications and shine light on gaps where opportunities are available.

In writing this book, we considered addressing privacy issues in a separate chapter to display our consideration of the more troublesome issues with Big-Data collection and use. We ultimately decided against this approach. Engineers and businesses too often consider privacy of their users or customers after they have almost fully conceived their initial product. The privacy features then become add-ons, affixed after the main design is mostly complete. We did not want our book to mimic this approach. We believe that peoples expectations of privacy and comfort level with sharing data should be considered from the beginning, baked deeply into any application or product design. Thus, throughout the book as needed, we reflect this sentiment by addressing issues of privacy, peoples knowledge of how data are collected and shared, their comfort with these approaches (with comfort often a moving target affected by many variables), and privacy-conscious ways to build applications.