Barry Libert - The Big Data Revolution

A business running without accurate data is running blind.
- Ash Mahmud

In the third century B.C., Egypts ruling Ptolemaic dynasty erected the Library of Alexandria. A testimony to civilization and learning - and the dynastys own wealth and influence - the library was reputed to hold the sum of humankinds knowledge. For the next twenty-three centuries, this body of knowledge grew at a steady pace and libraries were constructed all over the globe to contain it. Then in the late twentieth century the Internet came of age. This historic advance revolutionized information gathering and storage capabilities. Today, there is approximately 2.5 trillion times as much information in the world as was held in the Library of Alexandria - and the amount is doubling every three years. The vast majority of it is stored not in libraries but on servers. In 2000, three-quarters of the worlds stored information was in analog form; now, that figure is less than 2 percent. In other words, to explore the worlds knowledge a person no longer needs a library - a laptop or tablet or smartphone will do just fine. This vast digital storehouse is, in essence, big data.

Big data means big profits. Research from the Harvard Business School, the Massachusetts Institute of Technologys Sloan School of Management, McKinsey & Companys business technology office, and the University of Pennsylvanias Wharton School shows that businesses that make the most of big data are fully 6 percent more profitable than those that dont. In times like these, thats an uncommonly large number.

Heres what else you need to know about big data.

The Harvard Business Review concluded that the companies that significantly outperform their peers are more likely to collect multiple types of data - from that generated by RFID tags, for instance, to data from Web-tracking technologies. Furthermore, high-performing companies are more in touch with data than their less-successful rivals. When asked about marketing and communications, 59 percent of executives from top-rated companies called data extremely important to their businesses as compared to a 39-percent ranking from lower-rated concerns.

As for the technology itself, executives of companies that are ahead of their peers in analyzing data gave it high marks. Forty percent of high-performing companies said the speed at which their organization processes data has increased significantly over the past twelve months, and that speed is producing dividends.

A joint study conducted by IBM and MIT and published under the title, The New Intelligent Enterprise concluded that the number of businesses using analytics to build a competitive advantage had jumped by almost 60 percent from a year earlier. The study said that nearly six out of ten organizations are using analytics to differentiate themselves from competitors.

Case studies back up the results.

Picture a customer call center that has the technology to detect a change in tone as a frustrated customer raises his voice to say: This is the third power outage Ive had in one week! A big-data solution would both identify the words third and outage as negative terms affecting the consumer, and the tonal change would be another indicator that he or she might choose another provider. These insights can be gleaned from unstructured data. But what if that unstructured data could be combined with the customers record data and transaction history? Now the company has a personalized model of the consumers value and a sense of how tenuous the relationship has become.

The demand for big-data insights has risen so quickly that there are now numerous companies whose product itself is big data. Cases in point:

• Anyone who flies regularly knows it isnt always a pleasant experience. And one of the most frustrating parts is the all-too-common wait after landing for a gate and ground crew to be ready for deplaning. This happens when a plane lands early. If a plane lands late, its less annoying for the passengers, but the ground crew has been waiting, costing the airline money. Its not only the carriers that suffer; airports with lots of late and early arrivals develop bad reputations with passengers.

Enter PASSUR Aerospace, a big-data company that tracks plane arrivals with unprecedented accuracy. Its rightETA program relies on patented algorithms backed by over 150 passive radar sensors and integrated with real-time mining of multiple databases. Every 4.5 seconds, PASSUR collects data on every plane tracks. It also uses its vast storehouse of past arrivals to compare previous landings under the same conditions at the same airport. PASSUR currently has contracts with the top eight North American carriers, sixty U.S. airports, 200 corporate aviation departments, and the U.S. government.

• IBM has a patent on technology for securing premises using surface-based computing technology. The surface in question is the floor, and the technology identifies what or who is on the floor - i.e. furniture or people, what the object or person weighs, and when and where they move. In other words, IBM has developed smart floors. Walk into a room and the lights come on, and the latest episode of Mad Men appears on your flat-screen. If an elderly person falls, Lifeline will receive a signal. Stores, casinos, hotels, and government agencies could use the floors to gather data on foot-traffic patterns and use it to redesign their spaces. And it could be a potent antitheft device, alerting homeowners and businesses to an invader with unprecedented precision.
• New York City is using big data in innovative and effective ways. It was a pioneer in the use of proactive policing, which uses computers to track crime as it takes place in the citys neighborhoods. From experience, police know that a spike in minor crimes in a neighborhood is the precursor to a wave of more serious offenses. When the police departments big-data modeling shows this happening, officers flood the area. The citys crime rate has fallen to historic lows since this system was instituted. It works.

So does the citys use of big data to cut down both on fires and violations of the housing code.

With affordable housing at a premium in New York, many buildings and apartments are illegally subdivided into small rooms. Often these rooms are shared by groups of immigrants from Asia or Latin America who work at low-paying jobs. Fire-department data showed that fires were far more likely to occur in buildings that had been illegally subdivided. But with approximately 25,000 complaints a year about overcrowded buildings and only 200 inspectors, the city was overwhelmed. Big-data analysts in then-Mayor Michael Bloombergs office searched for ways to identify which of the 25,000 complaints were most egregious. They created a database of all 900,000 buildings in New York, along with information from nineteen city agencies that included ambulance visits, rodent infestations, calls to 911, unpaid taxes, unexplained spikes in utility usage, building age, neighborhood crime rates, and more. They then cross-referenced this with five years of fire-department information that ranked fires by their size and cause. The data quickly demonstrated that certain combinations of complaints, fines, and other violations were a reliable predictor of where fires would occur. Inspectors could now triage their visits. The results speak for themselves: Before the big-data initiative, inspectors issued emergency vacate orders to 13 percent of the buildings they visited; today that figure is 70 percent. And the numbers of fires in the city has decreased dramatically.

• Big data can not only prevent fires, it can save lives. Canadian medical researchers have developed a series of big-data measures to improve the survival rate of premature babies. The goal is to determine which babies are most likely to develop life-threatening infections. The process continuously measures sixteen vital signs, including breathing, blood-oxygen levels, organ function, blood pressure, and temperature. This information is aggregated into a 1,000-data-points-per second flow that alerts doctors to subtle changes that are often the precursor of infection. Prophylactic antibiotics can then be administered. The system has increased the survival rate of premature infants at the hospitals where it is used.