Cesar Perez Lopez - BIG DATA ANALYTICS AND HADOOP SOLUTIONS WITH SAS TOOLS

introduction

Big Data techniques are a new approach on how to store, manage and analyse all the data generated by an organisation for optimal decision making. This data is both structured and unstructured and comes from different sources such as e-mail, medical images, mobile devices, social networks, etc. In short, the origin of the data is all the new sources that have emerged in recent years. What we understand by Big Data, and all the associated technologies (Analytics), is to convert data into useful information for decision-making in any organisation. It is about achieving a proactive approach to the customer by means of analytical environments that are reliable and scalable, and allow new products, business models, etc. to be created. And all this entails adapting processes and technologies because data must be analysed in a different way.

Big Data can no longer be managed with traditional relational databases and business intelligence applications. Such management has been done for many years, but now requires a new approach. We have to change the business processes and relational databases we used to have. Because data now comes from many different sources, so new technologies have to be used to exploit it. The transactional analysis of data is now completely different, as are the technologies that capture, store, manage and retrieve it.

Big Data includes everything from data capture to storage, management, process and data analysis and querying of information. This requires managing different data models and scalability. Big Data implies a change of focus in technology, because significant scalability is needed; and in applications, because data is the centre of gravity.

All sectors are susceptible to using Big Data technology, from the financial sector, energy, mobile telephony, public administration, research, etc. For example, typical projects are those in financial institutions that, with information storage and analysis technology, have different customer data from different sources and transaction systems. This relative data must be intelligently grouped to allow segmentation in order to be able to offer the most appropriate products according to purchasing power, level of risk, investments, etc. The same is true for the Administration, which has a multitude of data and files, which can be cross-referenced between the different bodies, local and central, to improve service to the citizen. In short, companies and institutions seeking business agility need a way to analyse all this structured and unstructured data by properly applying Big Data and Analytics techniques and tools.

BIG DATA AND ANALYTICS

As mentioned in the introduction, todays data analysis requires the use of statistical techniques to learn from the data, from patterns and anomalies, from predictions and from professionals who know how to use them. The use of Big Data technologies not only allows us to increase processing capacity, but also to find those ideas that allow us to obtain the knowledge embedded in the data, as long as we have the profiles and experience to carry it out.

For this reason, Analytics techniques (essentially Data Mining and Business Intelligence) and Big Data go hand in hand for the optimal exploitation of information. Professionals, with skills in mathematics, statistics and computer engineering, who are able to extract the maximum value from the organisations data through Analytics, must work together with optimal Big Data infrastructures. The management and analysis of big data, structured and unstructured, applied in fields such as scientific research, health, security, social networks or media, among others, is a unique tool for companies to gain competitiveness and improve the life of citizens. This tool can only be optimised with the combined application of Analytics and Big Data techniques.

The following figure shows the process of interrelation between Big Data and Analytics techniques (essentially Business Intelligence and Data Mining).

The bottom left of the figure above shows the external sources from which information is extracted for our business. These sources are of a different nature, combining graphical information, textual information, information from social networks, information from non-relational databases and information from other sources. It is precisely the performance of these tasks that constitutes the essential part of Big Data. The information obtained from internal and external sources is stored appropriately and made available for analysis via the appropriate interface of the Big Data tools. The right-hand side of the figure shows how the information obtained by implementing Big Data techniques is processed by means of Analytics techniques such as Data Mining and Business Intelligence to obtain the knowledge stored in the data.

Given the great progress that exists every day in information technologies, organisations have had to face new challenges that allow them to analyse, discover and understand more than what their traditional tools report on their information. The need for Big Data arises at the same time as the great growth in recent years of the applications available on the internet (geo-referencing, social networks, etc.) that have been an important part of companies business decisions.

The concept of Big Data applies to all information that cannot be processed or analysed using traditional processes or tools. There are four key characteristics that define information related to Big Data:

Volume . Big data is produced in much larger quantities than traditional data. For example, a single jet engine can generate 10 TB of data in 30 minutes. With more than 25,000 airline flights per day, the daily volume of this single data source alone runs into petabytes. Smart meters and heavy industrial equipment such as oil refineries and drilling rigs generate similar volumes of data, compounding the problem.

Velocity . Social media data streams, while not as massive as machine-generated data, produce a huge influx of insights and valuable relationships for customer management. Even at 140 characters per tweet, the high velocity (or frequency) of Twitter data provides large volumes of information (over 8 TB per day).

Variety. Traditional data formats tend to be relatively well defined by a data schema. In contrast, non-traditional data formats exhibit a dizzying pace of change. As new services are added, new sensors deployed, or new marketing campaigns launched, new data types are needed to capture the resulting information.

Value . The economic value of different data varies significantly. There is usually good information embedded in a large, broader set of non-traditional data: The essential challenge is to identify the valuable information, transform it and extract the data for analysis. From the suitably extracted and transformed data, the knowledge contained in the data is analysed.

The Hadoop platform plays an important role in Big Data work. The importance of Hadoop lies basically in the fact that it allows very intensive massive computational tasks to be developed, dividing them into small pieces and distributing them over as large a set of machines as desired. The analysis is performed on petabytes of data, in distributed environments made up of many simple machines. This is a very reasonable value proposition in todays hyper-connected times, and is used to the hilt by companies such as Google, Yahoo!, Tuenti, Twitter, eBay and Facebook. The use of Hadoop is catching on fast in all kinds of companies .

The open source platform Hadoop is currently the leading tool for analysing large amounts of data.